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ABSTRACT 

A project vas conducted to assess the evaluation 
needs of post secondary cooperative education progr<aa administrators 
and to develop procedures and checklists for assessing productivity 
and impact of post secondary cooperative education* This vork built 
upon a general design developed in 1976 under part C of Public Lav 
90*-576 for iaproving planning, aanageaent and evaluation of 
cooperative education prograas. The evaluation coaponent of that 
djesign «as adapted to aeet the special needs of post-secondary 
cooperative education prograa directors to fccus cn aeasures of 
productivity and iapact, and to address the probleas of sex and 
ethnic fcias* The project involved the folloving: revien of relevant 
postseccndary cooperative education docuaents; studies and interviews 
vith local directors regarding evaluation needs; a revision of the 
evaluation design to reflect special needs of postseccndary 
cooperative education prograas and ^to focus cn aeasuring productivity 
and impact; the developaent of specific procedures and^ checklists to 
assess existing evaluation procedures and/or to develop new ones in 
critical checkpoint areas; and a pilot-test of products vith 
post-secondary cooperative education prograa adainistrators* A set of 
procedures and checklists for evaluating the productivity and impact 
of postseccndary cooperative education prograas resulted froa this 
study, (The checklists and instructions on their use are appended*) 
(Author/JB) 
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OBJECTIVES 

•3 ' 

The objectives of this project were: 1) to assess the evaluation needs of post 
secondary cooperation education program admin istrators and ; 2) to develop procedu 
and checklists for assessing [Productivity and impact of post-secondary co- 
operative education. 

PROCEDURES 

This work built upon a general design developed In 1976 under Part C for 
Improving planning, management and evaluation of cooperative education pro- 
grams. The evaluation component of that design was adapted to meet the 
special needs of post-secondary cooperative education program directors to 
focus on measures of productivity and impact, and to address the problems 
of sex and ethnic bias in cooperative education programs. Based on that 
design, procedures, and instrumjents were developed and tested. The project 
Involved: 1) revie\S of relevant post-secondary cooperative education documents, 
studies and interviews with local directors regarding evaluation needs; 2) 
revision of the evaluation design to reflect special needs of post-secondary 
cooperative education programs and to focus on measuring productivity and 
Impact; 3) development of specific procedures and checklists to assess 
existing evaluation procedures and/or to develop new ones tn critical check- 
point areas; k) a pilot-test of products with post-secondary cooperative 
education program administrators. 

EXPECTED CONTRIBUTION TO EDUCATION 

The results of this effort as contained in this report, includes a set of 
procedures and checklists for evaluating the productivity and impact of post- 
secondary cooperative education programs. As post-secondary cooperative 
education programs are increasingly being scrutinized in terms of impact 
and effectiveness, this evaluation manual will assist local project directors 
In evaluating their programs and increasing their program's productivity. 



CHAPTER I 
INTRODUCTION 

I . BACKGROUND 

Cooperative education programs at the post-secondary level are 
growing. In I969, only 120 post-secondary schools had cooperative 
education programs enrolling a total of 19,050 students- By 1975, 968 
post-secondary schools had such programs enrolling approximately 165,000 
students. Both public (70-9% of the schools reporting programs in 1975) 
and private (29.32), and junior {h6.3%) arid senior (53-7^) colleges were 
represented (Brown and Wilson,, 1975). Federal money allocated for post- 
secondary cooperative education programs has grown from $1,532,278 in 
1§70^to $10,750,000 in 1975 (DHEW, 1976). 

Continued growth of cooperative education programs seems extremely 
likely. President Ford voiced support for the expansion of work experience 
programs in his speech on^'^work and education'' at Ohio State University 
In August 197^. The recent evaluation of vocational education programs 
by the General Accounting Office took educators to task for under-uti- 
lizing work experience, stating, "It is generally acknowledged that 
inclusion of actual work experience in the vocational education curri- 
culum provides students with valuable real life exposure to work require- 
ments and helps assure that training is appropriate to employer needs" 
(Comptroller General, 197^). 

A recent national survey of the effectiveness of school-based training ^ 
programs stated: "If such programs are to be even more effective, though, 
they should provide students with more experience, more information about 
related jobs and up-to-date tools" (Benz, 1977). 

The National Manpower Institute, in its report. The Boundless Resource , 
released in November 1975, recommended the development of programs giving 
-all students at least 500 hours of work or service experience and the 
institution of a comprehensive program of comm'^nity internships and work 
apprenticeships (National Manpower Institute, 1975). 



For all tha support for the concept, however, there is a serious 
Impediment to the effective/productive expansion of post-secondary 
cooperative education programs. There is little knowledge as to what pro- 
gram components contribute to the success of these programs or of the rela- 
tionship between expenditures and resource utilization and outcome. Little 
real evaluation of these programs has teen done and there is a glaring lack 
of data on the differential impact of programs on women and minority group 
members. 

Recently, the problems of sex bias, sex discrimination, and sex 
stereotyping have come to the attention of educators, legislators, and 
social scientists (e.g. Steiger, 197^; Steiger and Cooper, 1975; Steiger 
and Szanton, 1976; Steele, 197^; and House of Representatives Report 
No. 9^-1085). A major cause for their concern was that women earn, on 
the average^ much iess than men (57% as much in 1973) (US Department 
of Labor, 1975). Reasons for this include discrimination against women 
and clustering of women in low paying jobs (Sweet, 197^). Women are 
restricted to traditionally female occupations, and these are at the 
bottom of the income scale (Sommc/s, 197^). Vocational education pro- 
grams reinforce these trends by training female students either for low 
paying, no status, traditionally female jobs or for a future as home- 
makers not in the labor force (Steiger and Cooper, 1975). The faulty 
assumption underlying these patterns is that women spend little, if 
any, time in the paid labor force or are not breadwinners. Yet today 
90% of women work at some time in their lives and the percentage employed 
in a given year has steadily increased from 28.2% in 19^0 to ^5.2% in 197^. 
Furthermore, many women are their family's only breadwinner today. 

The most common form of evaluation of cooperative education programs 
is a simple survey of the opinions of students, employers, and administra- 
tors regarding the success of the program. Yet in a recent study, f-rankel 
found no correlation between the rating administrator? gave their programs 
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and the satisfaction of students and employers. In fact, on some Indices 
high marks by administrators on program quality had negative correlations 
with successful outdomes as measured by FrankeK Furthermore, Frankel 
was restricted In his ability to analyze the relationships between acti- 
vities and outcomes because of the lack of follow-up data on students. 
He found that only (>]% of the programs had any follow-up data on students. 
He found that only 61% of the programs had any follow-up procedures at 
all, and the vast majority of those were the smaller programs. Only B% 
of the programs with 40-99 students had follow-up procedures and none 
of the programs with more than 100 students collected follow-up data 
(Frankel, 1973). 

The General Accounting Office report severely criticized all vocation 
a1 education programs for poor evaluation procedures. 'Mn the states we 
visited, the report states, "the existing vocational programs at all 
levels lacked adequate student follow-up. We were told that without 
this type of information; (1) it is extremely difficult to determine the 
extent to which specific training is impacting on Individual and labor 
market needs, and (2) essential information on which to base Instructional 
changes is not available to vocational educators and planners" (Comptroller 
General, 197^). 

There is some evidence that local administrators are receptive to 
this approach. Frankel found in his interviews of program administrators 
that they listed development of program objectives and design of valid 
follow-up procedures, both essential components of evaluation, among 
their major problems [Frankel , 1973)'. A 197^ project in Kentucky found 
that cooperative education program directors who lacked evaluation skills 
responded well to a training program (Comptroller General, 197^). 

The Education Amendments of 1976 included a number of provisions 
reflecting Congressional intent^for greater accountability through evalua- 
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tlon. For example, a new requtrement directs the Commissioner to give 
priority to programs which show the greatest promise of success. Evalua- 
tion should also address the extent to which programs in the academic 
discipline have had a favorable reception from employers. Additionally, 
programs which purport to teach entry-level job skills are to be evaluated 
by sampling techniques, If possi^i^'le, according to the extent to which pro- 
gram graduates find employment in related occupations and are considered 
well trained by their employers. While other provisions would indicate 
a gross expansion of cooperative education (e.g., fulltime employment 
Is no longer a requirement for students), there are clear indications 
that accountability through evaluation will increasingly become a higher 
priority in Congress. 

tl PURPOSE OF TH I S • PROJ ECT 

The purpose of this project is TO DEVELOP PROCEDURES AND INSTRUMENTS 
FOR ASSESSING THE PROCEDURES USED TO EVALU/^TE THE PRODUCTIVITY AND IMPACT 
OF POST-SECONDARY COOPERATIVE EDUCATION PROGRAMS. These procedures and 
Instruments are in the. form of checklists designed to be used.^by local 
program administrators to develop and/or improve, evaluation components 
In their programs. The procedures and checklists are based upon the 
specifications of/a general model for evaluating education programs de- 
signed by Education TURNKEY Systems, Inc., and Steiger, Fink, and Kose- 
coff. Inc. (SFX) as part of a 1975 Part C project and subsequent research 

conducted in 1976-77. 

TURNKEY and SFK feel that the problem of Inadequate evaluation de- 
sign and procedures to assess productivity must be solved at the local 
level. The procedures and instruments for assessing the productivity 
and Impact of post-secondary cooperative education programs must be use- 
ful to local prcgram administrators. Philosophically, we believe the 
accountabiUty for program effectiveness should rest with the person 
nearest to the program who is in a position to control its direction. 
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Practically, we believe that effective continuing evaluations of pro- 
grams, and utilization of evaluation data for the improvement of pro- 
grams and of resource allocation, can only be accomplished with the sup- 
port and commitment of the local program administrator. We also believe 
that the problem of sex bias in vocational education programs will only 
be solved when awareness is raised at the local level and the problem Is 
addressed by teachers, administrators, and counselors In all phases of 
their work. 

IN APPROACH TAKEN 

The major developmental activity of this project was the design and 
development of a set of checklists to be used to assess existing evalua- 
tion fomts and procedures and/or as the basis for the development of new 
evaluation forms and proceciures for critical aspects of cooperative edu- 
cation programs at the post-secondary level • The checklists and proce- 
dures were based upon assessment of needs Indicated through the review 
of literature and through personal discussions with a number of adminis- 
trators of post-secondary work study programs in the Washington, D.C. 
area and so elsewhere. 

The specific activities undertaken during the performance of this 
research and development effort are briefly described below. It should 
be noted that the delineation of specific tasks and activities differs 
scKTiewhat from that included in the original proposal, although all of the 
activities originally proposed were undertaken. There are several rea- 
sons for this recategorlzation. First, when the proposal was originally 
submitted, the 1975-76 research effort on which this project was based 
was only half complete. Hence, some of the changes In that project re- 
commended by USOE impacted upon the nature of the general model, which 
In turn had a ripple effect upon the nature of the proposed effort for 
the 1976-77 period. Second, after the proposal was funded, in a. meeting of 
Part C project directors in St. Louis in October 1376, a number of areas 
were emphasized as having relatively higher priorities for USOE officials 
than was stated in the original guidelines. And last, the Education 
Amendmen'ts of 1976, passed In October 1976, had a number of provisions 
as mentioned above, which impinge upon evaluation activities and require- 
ments related to post-secondary cooperative education programs. In anti- 
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cipatlon of Implementation of these provisions, the priority focus of the 
sfudy, again, changed somewh?*- However, It should be re-emphaslzed that 
all of the activities originally proposed were Indeed undertaken, even^ 
though priority areas within these activities changed somewhat. 

Task 1 - Refine Overall Management Plan 

During the initial stages of the project, the overall management 

plan was refined, and minor changes made* It was originally propo^^ed 

that field test efforts wouH be conducted. In Grand Rapids, Michigan,^ on 

the assumption that a post-secondary cooperative education program v«Duld 

be funded under Part D. When It was found that the Grr^nd Rapids pro|x>sal 

was indeed not funded, an alternative institution to be used for field 

testing was Identified, In light of the- good cooperation of the District 

of Columbia (SEA level) and original contacts with local post-secon|ary 

institutions. It was decided that the field test ingj would be conducted 

with the Washington Technical Institute (WTI) and, to some extent, with 

Federal City College (FCC), Also, unlike the previous year's effort] 

t 

during which states were requested to nominate exemplary planning anc| 
evaluation models, it was decided to focus more specifically on a vej'y 
limited number of exemplary models for potential revision and appl Icitlon, 
Also, It was decided that certain portions of the model described Injthe 
final report on the 1975-76 effort could be adapted without much chapge 
for application In descriptive aspects of this year's effort, demonstrating 
the attempt to build upon the previous year's effort In developing sj^e- 
clflc checklists and components for the post-secondary level, Flnaljly, 
based upon discusslpns subsequent to the October St, Louis meeting df 
project directors for Part C Research Projects, It was decided that |the 
field test would focus only upon fact validity In light of the limited 
number of Institutions and the nature of the Instruments themselves 



described below. 



Task 2 - Conduct Needs Assessment 



as 



In conducting the needs assessment of program administrators cjf post- 
secondary cooperative education programs, a number of concurrent activities 
were undertaken. First, exemplary models and procedures Identified In 
last year's effort were again reviewed for potential application and revi- 
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ston for this year*s effort. In addition, a number of reports, parti- 
cularly those of an evaluation nature, provided some new Insights. Se- 
cond » a substantially greater level of effort was devoted tc^ discussions 
with USOE officials Involved In post-secondary cooperative education In 
an attempt to Identify perceived needs on their part, especially with re- 
ference to the Education Amendments of 1976 and their Implications for 
evaluation of cooperative education programs at the post-secondary level. 
Numerous USOE officials provided extremely useful insights as well as 
projections of their needs over the next two years. 

Since the major focus was upon project administrators at the local 
level,, 'the project team developed a list of topical areas which formed 
the basis for project team Interviews with various individuals in post- 
secondary institutions involved in cooperative educ9|tion programs. The 
levels of individuals interviewed included the President of tihe college 
or Institute, the individual responsible for overall instruction, the 
Director of Cooperative Education Programs, and numerous officials respon- 
sible for various related functions such as Job Development, Placement, 
Counseling"';' and so on. The specific topics addressed In these Interviews 
are 1 1s ted In Tcble 1 . 

" : Table 1: TOPICAL AREAS FOR INTERVIEWING ~" 

• Perceived productivity of programs and the measures used to assess 
productivity over time. 

• Perceived Importance of various criteria from various perspectives. 

• Problems Inherent In associating cost with program effectiveness. 

• Type of cost reporting system currently used for reporting* 

• Hypothetical and Ideal Information reporting system to be used for 
evaluation purposes. 

m Strengths and weaknesses of existing reporting evaluation systems. 

• Anticipated evaluation problems over the next two years. 



As a result of Identifying evaluation needs and problems through the 
above iapproaches, a hypothetical model of evaluation checkpoints In typi- 
cal post-secondary cooperative education programs was developed.. This mo- 
del and the evaluation checkpoints are displayed In Exhibit 1-1. The pro- 
ject team then Identified the critical priority evaluation checkpoints for 
/hich Instruments would be developed based on the interviews and needs 
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assessment and a review of the Education Amendments of 1976. From this 
discrepancy- type analysis, priority Instruments included In subsequent 
chapters were developed.- 

Task 3 - Revise General Model to Meet Post-Secondary Needs 
Concurrent with the needs assessment, the project team reviewed the 
overall general model developed In the prior year to determine which as- 
pects of It would be generally applicable. It was felt that a number of 
the checklists that had been previously developed could be refined for^ 
application In this effort and specifically, that previous work conducted 
In planning an evaluation design would be- extremely appropriate based 
on discussions with officials at WTI and FCC. The results and findings 
of this study and analysis effort, deliberations, and discussions are 
summarized In the following section, entitled Evaluation Issues. ^ 

Task k - Develop Specific Checklists 

Rather than attempt to develop specific evaluation forms and tech- 
niques for high priority checkpoints described in Exhibit !-t, jt was 
decided that checklists would be developed to serve two purposes: 

• to assist project administrators In assessing their exist- 
ing information system or evaluation components related to 
Identified checkpoints; or 

• to assist project administrators in developing such evalua- 
tion forms and instruments where they did not already exist. 

Developed between February and May (as described In a subsequent chapter), 
these checklists were field-tested for fact validity by having a number 
of administrators of cooperative education programs review the Instruments 
In light of their perceived needs. After personal interviews with pro- 
ject administrators following their reviews of these checklists, the In- 
struments were revised as necessary. 

In the next section, we discuss some of the major evaluation Issues 
and priorities which we Identified during the needs assessment process and 
Interviews. In subsequent chapters, the specific priority issues which 
are addressed in the checkl isf>|!|re described In greater detail. The sum- 
mary of findings presented in th4 next section Is diagramed In Exhibit 1-1. 
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IV NEEDS ASSESSMENT; FINDINGS 

ES^aluatton Checkpoints - Student Flow 

A number of evaluation issues relate to certain evaluation checlc- 
polnts which occur from the time the student enrolls in the program to - 
the time when the student leaves the program for continuing education 
or employment. These issues are summarized below. 

1. Counseling: First, prior to enrolling in a specific 
cooperative education program, a student usually receives formal or in- 
formal counseling from institutional staff or referral agencies (e.g., 
Veterans Administration, CETA, etc.). In the initial step toward course 
selection, a number of considerations constrain the student/course 
matching process. For example, offerings by the institution may be limi- 
ted due to a number of factors: 

• the type of employers in the area; 

• the amount of personnel available for counseling compared 
to the potential student enrollment; 

• the availability of sicilled labor force members who are 
unemployed in the specific geographical area; and 

• other factors beyond the control of the program administra- 
tor and even the institution, in certain cases. ( 

On the other hand, there are a number of issues which can be ad- 
dressed in xin effective counseling program for which procedural checlclists 
would ensure coverage and potential solution. The first area is the eli- 
^ mlnation of sex stereotyping and bias, which is clearly a major objective 
of. the Education Amendments of 1976 both in vocational . education and post- 
secondary education programs. The second area, reflected again in the 
Education Amendments of 1376, is the priority funding gWen to cooperative 
education programs which show the greatest promise of success- In order 
- for post-secondary institutions to meet these provisions, evaluation of 
existing counseling practices must be undertalcen. It would appear that 
these areas lend themselves to checlclist type instruments which can be 
used by program administrators in meeting many of these program require- 
ments . 

2. Prescription: A second area of focus is the diagnostic/ 
prescriptive/job analysis process as applied to an individual student as 
he enrolls in a cooperative education program. While a number of the 
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constraints Impinging upon effective counseling are in force at this 
checkpoint, there are perhaps a number of problem areas over which ad- 
ministrators could have more control given the availability of trained 
staff. Diagnosis, prescription, and job analysis are highly Interre- 
lated. A key Issue here Is whether the program administrator at the 
local level should begin with the job analyses or with the diagnosis 
of Individual aptitudes and Interests, and so on, a questloa usually 
answered by the overall philosophy of the cooperative education program 
within the Institution. The program administrator at one of the more 
Innovative and successful Institutions suggested that the question really 
Is- not one of elther-or. For example, this Institution has a very hard- 
nosed ^business- 1 Ike approach to cooperative education, with the only cri- 
teria for success being the number of jobs held or the number of partlcl- 
pants and non-partlclpants at entry level. On the other hand, the admini- 
strator' has been able to take Into account Individual aptitude and Interest 
levels by finding positions for participants as far as 2,000 miles away.^ 

The type of' job structuring Is also a related Issue of Initial fo- 
cus: either the student or the employer. Under the 1976 Amendments , 
funds are now available to Institutions to provide job development and 
job structuring services working with employers to ensure that the par- 
ticipant Is being trained in a career ladder rather than being used by 
the employer for a short period of time with limited growth and vertical 
mobHlty opportunities. There would appear to be s very strong need on 
the part of project administrators to have checklists which allow them to 
review existing procedures In this area to ensure more effective training, 
higher probability of vertical and even horizontal mobility upon comple- 
tion, and'greater student flexibility (within limits) to maintain Interest 
levels. I 

Monitoring: Once the student enrolls In the Instructional 
cooperative education program, evaluation activities through student moni- 
toring should focus upon a number of Issues, some of which have been men- 
tioned earlier. One such Issue Is the degree to which the Initial dlai>- 
nosls and prescription regarding the learning profile of the student Is 
matched with the Instructional methods and techniques, which are used 
both In the academic and on-the-job environments. Another Issue Is the 
degree to which content, In both the academic and to a lesser extent on- 
the-job training environments, Is matched wi h actual job requirements. 
One of the evaluation questions raised In the Education Amendments of 1976 
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\z whether the success of a cooperative education program Is due to a 
heavy commitment by the Institution Itself, based upon funding and 
other Irrcentlves Available to that Institution, or whether It Is directly 
related to a favorable reception on the part of employers who train and 
then aiiploy participants upon completion. 

Evaluation Checkpoints - Program Level 

Many of the evaluation Issues as perceived from a student point of 

« " 

view also arise as one views the overall program. There do exist, how- 
ever, a number of higher level Issues which should be addressed by any 
evaluation system used in cooperative education programs-. 

The first Issue is the conception of the program Itself. In a num- 
ber of Institutions, the cooperative education program Is that which occurs 
In the employer's environment. However,, In some of the more Innovative 
and successful institutions, the cooperative education program was defined 
as all of thev components and services provided to the Individual,? ranging 
from counseling through placement on the job. In sych Instances, the key 
to success appears to be the degree of coordination among the various of- 
fices within an institution responsible for such functions. Indeed, the 
role described by one successful program administrator was that^ of essen- 
tially a "coordinating unit and ombudsman" for the student. From an eval- 
uation point of view, however, what constitutes a cooperative education pro 
gram Is' often dictated by funding sources, local or state statutes, and 
other factors which create inherent dif f icul f <es for anyone attempting to 
evaluate a cooperative education program, especially as one attempts to 
associate process variables with success or lack of success. 

The second area, and most critical from an evaluation point of view. 
Is the statement of program objectives, which in turn should be based upon 
an assessment of student, employer, and Institution needs. For a number 
of reasons, cooperative education program objectives are usually very glo- 
bal In nature rather than being specific. This can be attributed to a 
number' of factors, ranging from concern over being held accountable, to 
allowing funding'flexibil ity, to in some Instances purf^oseful ambiguity. 
Without clearly stated performance-based objectives. It Is difficult If 
not Impossible to conduct impact evaluations; without clearly delineated 
-process or Implementation objectives it is difficult to identify manage- 
ment changes which will result In more efficient operations; without ob- 
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jective assessment of needs, clearly stated objectl^'es may be irrele- 
vant, unrcaUstIc, and of little practical utility to participants- 
Assuming that objectives are based upon defined needs and are rea- 
listic, a program' evaluation Issue is the relationship of the Instruc- 
tional process in the program to the probability of achieving the objec- 
tives. Many of the student-related issues related to profile/matching 
method and content/ job requirement are appropriately re-emphaslzed here. 
Another Issue Is the proper mix of academic and on-the-job training, 

A third Issue, from ah evaluation point of view. Is the degree to 
which the management plan, as stated. Is actually being Implemented, In 
most cooperative education programs, especially those which have existed 
for .several years or longer, the "standard operating procedure" during 
the operational phase Is seldom described on paper. This Is of particular 
Interest since discussions with, numerous project administrators Indicate 
that a detailed management plan for planning, a new cooperative education 
program Is Indeed critical, and that within such a plan the degree of coor- 
dination among the various' of flees within the Institution and between the 
^ Institution and the employer Is particularly critical and appears to be 
highly associated with successful programs. Successful program adminis- 
trators repeatedly^ mentioned their role as a coordinator between the 
academic faculty and the employer Implementing cooperative education pro- 
grams. 

Last, from the evaluation point of view, the major task in assessing 
the program is to relate objectives to actual attainment and attainment 
to cost. Discussions with numerous project administrators indicate that 
virtually all have some measure of success which they relate to costs -even 
though the specific measures may reflect more the philosophy of the Insti- 
tution or the cooperative education program administrator rather than sta- 
ted objectives and official documents. For example, the Institution men- 
tioned earlier which has a very business-like approach to assessing its 
success would not be interested in an evaluation model designed for an 
Institution with a different philosophy. To Illustrate, the cost provided 
by the employer is indeed a real cost to the operations of the cooperative 
education program; however, the above institution would not Include these 
costs since it would argue that the employer, acting as a rational 
"economic man" would not be involved In the program unless the benefits 
to the firm were equal to or greater than the costs. And as described 
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later, the type of funding source often determines the perception of real 
cost Involved In a program. For example, outside funding assistance based 
on the number of participants would conjure up a different perception of 
cost from a cooperative education program which Is solely funded by the 
Institution based on a program requirement rather than the number of In- 
dividual participants. To be useful to the local program administrator, 
the evaluation design therefore has to be related directly to his per- 
ceived needs, which are a direct reflection of the philosophy underlying 
the program and a number of other factors Impinging upon it. 

Summary of Procedures (Chapter III) and Checklists (Chapters IV, V, VI) 

In this section we summarize the overall set of procedures and In- 
structional materials related to this development effort* Before doing ^ 
so It should be noted that this set of procedures and checklists reflects 
the general character istics .of the model developed during- the 1975-76 
period. 

Due to the wide variance In cooperative education programs at the 
post-secondary level, their philosophies, and funding sources, a decision 
had to be made regarding the types of programs for . which the set of 
procedures and checklists would probably be most useful. While we feel 
that these procedures will be useful to some extent to any cooperative 
education program, they are primarily designed for programs which have 
many of the characteristics listed below. These characteristics. Inci- 
dentally, appear to be correlates of successful or extremely promising 
Institutional cooperative educational programs. The "procedures and 
checklists, then, are. primarily designed for: 

c programs In which the administrators focus primarily upon 
the individual participant whose Individual success is the 
primary measure of overall program success; 

• programs in which evaluation appears to have a higher prio- 
rity than in more traditional cooperative education programs 
where the maintenance of the program Is an explicit or Im- 
plicit Institutional goal ; 

• programs In Institutions which follow an ^'economic" approach 
In designing and operating cooperative education programs; 

programs In which the program administrator's role Is perceived to 
be relatively broad, with emphasis on planning and coordina- 
tion and where participatory management and early Involve- 
ment of academic faculty members and employers In the plan- 
ning process can occur'. 



Trom an evaluation perspective, the implications of this focus are 
significant and could result In tighter designs. For example, performance 
criteria v^uld Include measures such as percentage of placement, attri- 
tion on the job after six months, and salary differentials comparing par- 
ticlpants to non-participants* On the cost side, employer resources con- 
sumed is only a secondary consideration since the economic just I fj cat ion 
for employee participation is that benefits are equal to or outweigh the 
cost. 

The results of this developmental effort and the resulting set of 
procedures and checklists are presented below, summarized by chapter. 

Review of the relevant literature on evaluation models is presented in 
Chapter 11. The discussion in Chapter ill focuses upon, the topic of 

selecting an evaluation design, building upon the general model developed 

/ 

in 1975*76 by the project team. The topics and checklists focus upon a 
number of critical areas as follows. First, selecting evaluation questions 
requires an iterative process of prioritizing the types of questions key- 
decision-makers wish to have answered and to negotiating among the various 
users, since all questions can never^be addressed In a program evaluation 
design. It is primarily the function of the program administrator to 
determine the' eval uat ion objectives or questions to be addressed, especially 
when a third party actually conducts the evaluation. 

Second, the factors to be considered in organizing the information 
collection process are covered in the checklist with subsequent discus- 
sions on factors and considerations, in actual- data collection; Primary 
consideration Is given to such factors as the accuracy of information, 
the reliability of the information', and to some extent 'the validity of 
data which is collected. Unless great care is taken during the data 
collection planning phase, subsequent evaluation activities are seriously 
hindered. 

third, planning and implementing data analysis and reporting is a 
third major focus of Chapter III. The major factors to be considered 
as these activities are undertaken are emphasized. The analysis plan 
should focus upon the primary questions to be answered, given the de- 
sired confidence level and general izeabi 1 Ity of the findings. Relatively 
simply statistical treatments useful In interpreting findings are covered. 
The primary emphasis in reporting focuses upon timeliness of information 



and the utility for dv^clslon-making, primariJy at the local level but 
also at higher levels^ h. eluding funding agencies. 

In Chapters IV, V, and VI, three problem-oriented evaluation check- 
lists arc presented, prefaced by discussions of the major Issues related 
to the checklists and the rationale for their presentation. The selection 
of the specific checklists was based upon an assessment of the evaluation 
checkpoints described in Exhibit 1-1 which as the reader will recall, 
resulted from the needs assessment and discussions with local program ad- 
ministrators of their program needs. The checklists are d*;,slgned to 
assist program administrators In those areas where evaluation procedures 
and forms are presently being used or in those instances where they are 
non-existent. In the latter case, forms and Instruments can be developed 
at a general level from the checklist. The areas on which checklists fo- 
cus are: (1) the counseling process, including diagnosis and prescription; 
(2) matching the academic content to the^job requirement and student pro- 
gress monitoring; and (3) cost-performance relationships. .;\ 

Throughout this document, a number of priority issues are addressed'^ 
at all major checkpoints in the overall evaluation model and sets' of pro- 

c 

cedures, including: sex-bias stereotyping; vertical and horizontal mo- 
bility considerations; Individualization of programs; and "output** or 
**impact** evaluation. 
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CHAPTER II 

BACKGROUND: EVALUATION OF COOPERATIVE EDUCATION 

During the past decade, evaluation has emerged as a social science 
discipline. As the Stanford Evaluation Consortium recently noted, evalua- 
tion lis an area of scholarship which has finally cqme of age. Evaluation, 
when applied to education programs. Is a set of procedures used .to appraise 
program merit and proNAde lYiformation about program goals, activities, and Impact 
and productivity (Fink and Kosecoff, 1977). The Importance of evalu';*"ing 
programs In education has been acknowledged by theorists and practitioners, 
ar.d most federally funded education programs contain provisions for evalua- 
tlon. Vocational and cooperative education programs are no exception. As 
noted earlier, the Education Amendments of I976 (P;L, o94-482,- Section 11) 
require that each State shall evaluate the effectives? of each program - 
bejng funded under the law, and that the results of the evaluations be 
used to revise the State's programs. P.Lj 9^-^82 further refcjulres that 
each State must evaluate, '*by using data collected, whenever possible, by 
statistically valid sampling techniques,** each program that purports to 

**lmpart entry level job skills according to the extent to which program 

I 

completers and leavers: 

"(i) find employment in occupations related to their training, 
ana 

"(il) are considered by their employers to be well-trained and 
prepared for employment...*' 

It is important to point out that P.L. 9^-^82 emphasizes program 
evaluation. This new focus may require a change In thinking for many co- 
operative educators who will have to turn their attention away from only 
evaluating students, faculty, or employers. For instance, instead of just 
providing Information about how well students are trained or how satis- 
fled employers are with their performance, cooperative educators are cur- 
rently required under P.L. 9^"^82 to produce data concerning the components 
of their programs that have the most impact on student performance and 
employer satisfaction. As noted earlier, evaluation efforts In cooperative 
education have tended to be minimal and unsuccessful (Blaschke and Stelger, 
1976) . 

The inadequacy of cooperative education evaluation efforts in post- 



id 

ERIC 



22 
-17- 



secondary schools was also emphasized by Cohen and Solmon (1976) whose 
study encompassed 1^-000 people, 2,000 documents, ^nd 96 campuses* Ac- 
cording to them, evaluations of ''cooperative education have not typi- 
cally been well done J' 

I CURRENT EVALUATION THEORY AND MODELS 

Host evaluation theorists and practitioners agree that the purpose 
of evaluation is to provide information about a program's impact, produc- 
tivity, and merit. However, approaches to gathering and reporting the 
Information vary. Steele (1976) describes three major evaluation models. 
The first, the ''Student Outcome" model. Is designed to, determine how well 
a specific number of students achieve predetermined objectives as a re- 
sult of their participation in a specific program. The second, the "pro- 
gram process" model, is used to estimate the degree to which certain 
program processes, like events or activities, are associated with student 
achievement or other possible program outcomes. Steele's third model, 
"policy" evaluations, is designed to assess the results to existing po- 
licles and programs and to test. old ones. 

The Center for the Study of Evaluation (CSE) at UCLA has a five-stage 
evaluation model. According to CSE, evaluation involves ascertaining 
the decision areas of concern, and then selecting, collecting, analyzing, 
and reporting information to the appropriate audiences (Klein, Fensterma- 
cher, and Alkin, 1971)- In CSE's model, a program should first be subject 
to a "needs assessment" in order to arrive at valid and consistent goals. 
Next, the program planners must develop a program that is designed to meet 
the goals. During CSE's third stage, the evaluator determines How well 
the program is implemented; while in the fourth, the progress of the pro- 
gram in meeting its goals is assessed. In the fifth stage of the model, 
the evaluator appraises the overall success of the program. 

Another major evaluation approach is the CIPP model developed by 
Stufflebeam (1968). In this model, there are four major evaluation stra- 
tegies: context, input, process, and product. The purpose of context 
evaluation is to identify needs and delineate problems underlying the 
needs. Input evaluation, the second strategy, is performed to identify 
and assess system capabilities, available input techniques, and designs 
for implementing them. Process evaluations result in the identification 



or production of defects in the procedural design or its Implementation 
and documentation of procedural events. The fourth strategy is product 
evaluation, and its purpose Is to relate outcome information to objectives 
and to context., input, and process information. 

Goal-free evaluation, suggested by Scrlven,. Involves the evaluator 
In assessing a program's impact and productivity without advance know- 
ledge of its goals. Scriven asserts that It Is the evaluator's respon- 
sibility to appraise a program's effects and not its Intentions. 

Still another model has been proposed by Stake (1967). According to 
the model, the two basic acts of evaluation are description and judgement, 
both of which are essential if education programs are to be understood. 
To wiploy descriptive judgement, the evaluator shouH uie data banks docu- 
menting information on ontecedent conditions, transactions, and intents^ 
as well as goals and objectives. 

Because of the relative newness of evaluation as a discipline, there 
Is a paucity of trained evaluators in cooperative education, and it. Is 
only recently that program managers hav-e accepted the need to study evalua- 
tion, to develop skills in conducting the follow-up research to determine 
what happens to ''program completers and leavers**, and to devise "statis- 
tically valid sampling techniques". In fsct, post-secondary school coop- 
erative educators in search of specific information about evaluation prac- 
tices will inevitably be disappointed. The Cooperative Education Informa- 
tion Clearinghouse at the Cooperative Education Research Center, Northeas- 
tern University (1975), for example, provides an extensive bibliography 
of articles and reports for cooperative educators. However, it has no 
special evaluation category, al^ough a "Planning and Implementation" 
classification of documents is available. To complicate matters, coopera- 
tive education ^programs have unique needs that make it clifficult to. auto- • 
mati-cally adopt existing general evaluation models or theories. The authors 
of the Cooperative Education Study (1975) write: "There appears to be no' 
universal or guiding principle — beyond that of meshing classroom and work 
experience in post-secondary education ~ which could serve as a single 
foundation for program evaluation. 

This report was, therefore, prepared as a vehicle for providing post- 
secondary school cooperative educators with the evaluation skills and in- 
formation they are currently unable to obtain from any tfther sources. 
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The approach to program evaluation described in the report was specifi- 
cally designed for cooperative education programs and is but It on the 
assumption that each one has unique features and guiding principles. 

li COOPERATIVE EDUCATION EVALUATION REgUIREMENTS 

Cooperative education needs an evaluation model which can be adapted 
to fit' the needs of a variety of program types and program settings. The 
model must be equally useful for distributive education, office occupa- 
tions, Industrial and trade occupations programs* and other subjv.ct areas 
and combinations of subject areas; for large cities and rural areas; for 
secondary and post-secondary schools; for standard classes and classes 

for students with special needs. The model must allow for Input on evalua- 

*■ • 

tlon questions from all concerned groups; students, parent'i, t^sachers, - 
administrators, employers, labor representatives, community groups, and* 
advisory councils. It must be sufficiently straight-forward and practical 
such that cooperative education program directors with limited resources 
at their disposal will find it realistic and useful. Yet,, it must also 
provide for the use of sufficiently sophisticated techniques so that evalua 
tlon questions concerning the progress and outcome of instruction, and 
the relationship of program components to outcomes, can be answered. 

\^Any effective procedure for the evaluation of cooperative education, 
such a^s the general model on which checklists were developed, should have 
the fol*1owing four features that distinguish it from other models and' 
make it especially appropriate to cooperative education programs: 

1. . Responsiveness to a program's needs. The evaluation 

model should be specifically designed so that it can 
be molded to fit the requirements of every program no 
^TiP^tter how unique they are. In -this way, the shape of ' 
ail evaluation is entirely dependent upon the purposes 
and nature of the program and is never imposed on it." 

2. Provisions for checks and balances. Because of the^ 
importance of the audience of ttie evaluation, each 
major category of evaluation activity should begin and 
end with reference to previous activities and to the 
need for consultation with the client to ensure that 
her/his needs are being mef. 

3. Action and practical orientation. The model should be 
developed so that the evaluation can provide timely^ 
relevant, and accurate information that can readily be 
used. This can be done by providing the framework for 
including the client in the formation of the evaluation 
and in the monitoring of its progress and quality. 



k. Attention to Issues of special concern to cooperative 
educators. A major goal of many cooperative education 
programs is the provision of skills and training for 
women and minority students. The model should incor- 
porate within It special attention to Important prob- 
lems faced by these groups, 1 lj<:e sex or ethnic bias In 
career Interest Inventories. The model should also pro- 
vide a framework for designing followrup studies (as 
required by f.L. 94-^82) even in situations that are 
characterized by high rates of student transiency and 
mobll Ity. 

• evaluation Is defined as *'a set of procedures used to appraise a 
program's merit and to provide Information about the program's goals, 
activities, impact, and piroductlvl ty**. (SFK 1975) There are two contexts in which 
evaluations can be conducted. In the first context, an evaluation Is 
conducted to Improve a program and the evaluation's clients are typically 
the program's orQanir^ers and staff, jn the second context, an evaluation • " 
Is conducted to certify the effectiveness of .a program; here the ev*.iua- 
tionU clients are typically funding agencies and Congress. 

The context for an evaluation is determined by the Information needs 
o^ the Individuals and agencies who must use the evaluation Informatfon, 
An^ evaluation Is performed Jn an improvement context when the evaluation's 
cllents^are concerned with Vinding out precisely where a change would make ^ • 

the program better. Usually, the organizers of a stl 1 1 -developing coopera- 
tive education program require this kind of information so that they can 
modify and iqiprove the program. On the other hand, an evaluation Is con- ; 
ducted In an effectiveness context when the evaluation's clients are par- 
ticularly concerned with determining the extent to which the program's 
overall quality can be guaranteed. Individuals who funded program develop- 
ment or who are Interested in using the program require this kind of Infor- 
matlon about a completed program's impact ana productivity. In addition, 
In an effect I veness context, the evaluator frequenty assumes a more glo- 
bal and Independent perspective than in an improvement context • 

The evaluation activities are organized into six major categories: 

1. Selecting Evaluation Questions^, 

2. Organizing Information Col Vectlon; 

3. Collecting t-nformat ion; ^ 
If. Planning and Implementing Data Analysis; \ 

5. Reporting Information; and 

6. Managing Evaluation Activities. ( 
In Chapter 1 1 1 of this report, each of- these activities Is described 

In detail. 2G 
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CHAPTER III 

COOPERATIVE EDUCATION EVALUATION IMPLEMENTATION PROCEDURES 



In this chapter, the procedures for conducting an evaluation of 
the productivity and impact of post-secondary coojaerative education and 
work experience programs are detailed In such a way as to provide the 
foundation for the development and use of the evaluation\i5hecknsts de- 
scribed In Chapters IV, V, and VL This chapter Is divided Into six 
sections, each representing a major evaluation category: 

• Selecting Evaluation Questions; 

• Organizing Information Collection; 

• Collecting Information; 

• Planning and Implementing Data Analysis; 

• Reporting Information; and 

• Managing Evaluation Activities, 
Each evaluation category Includes the following: 

• Introduction; 

• Considerations involved in performing specific evaluation 
activities; ^ 

• Problems 'associated with specif ic evaluation activities; 
and ^ 

• Case Example that Illustrates the application of the conside 
tlons and the effects of the problems. 
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I SELECTION OF EVALUATION QUESHONS 

An effective evaluation results in timely and believeable Information 
that Is useful In Improving or certifying a program. To ensure an eva- 
luatlon.^s effectiveness, the evaluator must pose questions that are res- 
ponsive to the needs of all concerned individuals. In formulating the 
questions, the evaluator must review the program's goals and activities 
and ascertain the kinds of information that will be acceptable as evidence 
of program success. 

Considerations Involved In Successfully Selecting Evaluation Questions 
To ensure a credible evaluation, the evaluator must: 

' 1. Review the program's goals and activities: When reviewing 
a program's goals and activities,^ the evaluator should become familiar 
with the interests and concerns of all groups who have a stake In the pro- 
gram and its evaluation: e.g., students, employers, teachers, administra- 
tors, or funding agencies. 

2. Be responsive to the types of information that will b6 con- 
vincing as evidence of the program's success: There are many different 
ways to prove that a program has Wen successful: e.g., that Its goals 
have been achieved, that it was managed so as to have no negative effects. 
Some ways include records of successful placements of students In jobs, 
successful performance of students on test of skills, and testimony of 
program graduates, employers, and parents. 

^ * 3. Pose specific questions that the evaluation's audiences 

want answered: Evaluation questions can take the following forms: 

• To what extent were- the program's goals achieved? 

• Were the program's activities Implemented as planned? 

• How effective were these activities in achieving the goals? 

• For which groups was the program most/least successful? 

• What did the program cpst? 

• How well was the program managed? 

• How did external and internal social and political forces 
influence the program's development and Impact? 

• What social and< pol i tical effects did the program have on 
the environment In which it was implemented? 

k. Hake sure that those participating in the evaluation under- 
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stand the procedures and products of the eva^luatlon: The evaluator must 
make sure that appropriate participants understand what an evaluation Is, 
the reasons for conducting the particular evaluation, how evaluation in- 
formation will be used, and that necessary releases for use of Information 
have been obtained from students or their parents* 

Problems In Selecting Evaluation (juef tions 

1^ The evaluator may have difficulty In obtaining the cooperation c 
of' participants. 

2. ^ Access to program documents or staff for evaluation purposes can be 
limited or not possible because of privacy regulations. 

0 c 

3. The program* goals may have been unclear or not measurably sta- 
ted, and/or program activities may be Impricisely described. 

k. Different Interest groups might be unable or unwilling to agree 
on the- evaluation questions and on what constitute as evidence of program 
success. 

5. The evaluation questions might not be on target because": 

• there are too many/ too few questions; or 

• they do not lend themselves to adequate answers, given the 
time and money available! 

A Case Example 

An evaluator was hired for a cooperative education program in an 
Inner-city community college. The college-was five years old, and its, 
staff and students were enthusiastic about the success the school had 
already achieved In placing students in a wide variety of jobs. The eval- 
uator began his work by identifying a list of questions that appeared to 
be relevant to students and administrators. To be sure his expectations 
were correct, all cooperative education- administrators and a large sample 
of students were shown the list of questions, and were asked to help re- 
fine them. At the conclusion of the review process, the evaluator felt 
certain that the results of the evaluation would be pertinent to all in- 
dividuals Involved in the program. Among the evaluation questions were: 

• How many students were placed in jobs directly related to 
their training? 

• Were there differences between men^s and women ^s satisfac- 
tion with respect to their training? 

• How much did it cost to provide training for each student 
who was placed In a job directly related to his/her training? 
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The cvaluator submitted two reports: an Interim report and a final 
report in June. Reaction to the Interim report came from only a few admin- 
istrators, but It was favorable. Reaction to the final report came from 
aPwlde variety of people, and it. was somewhat unfavorable. Employers com- 
plained that the questions asked by the evaluation ignored some of their 
needs and problems, such as whether they were satisfied with their role in 
the program. Faculty also Indicated discontent with the scope of evalua- 
tion questions. The situation described In this case example could 
have been avoided by initially submitting questions to or eliciting suggestions 
from all elements of the evaluation audience, i.e., faculty, students, 
administrators, employers... 

.11 ORGANIZING INFORMATION COLLECTION 

Planning Information collection activities for an evaluation Involves 
consideration of the evaluation questions, thfe Information collection* 
techniques, and the design strategy used to group and sample participants 
and to structure the data analysis. 

Considerations involved in Successful ly Organizing for Information. 
Collection 

1. Techniques used to collect evaluation Information: There are a 
variety of techniques that can be used to collect evaluation Information, 
Including Interviews, questionnaires, rating scales, observations, record 
reviews, and achievement tests. Each has advantages and disadvantages, 
and the evaluator must determine which will yield the most reliable and 
valid Information, given the inevitable constraints of time and money. 

,2. . Design strategies used to group and sample participants: Fre- 
quently used design strategies for cooperative education programs should 
Include case study designs, time series designs that compare the project 
population's present -^scores with those of previous year, and comparison 
group designs that include control groups, comparisons of gains when stu- 
, dents have a high involvement with the project activities with, the gains 
achieved when they have a low level of Involvement. 

3. Sampling: Sampling guides the selection of persons to be used 
In the evaluation and the assignmient of these persons to groups. The 
evaluator must determine whether some or all eligible students, teachers, 
employers, administrators, or advisory committee members will be included 
In the evaluation, and whether or not they will part iclpate in the new or 
traditional program. 
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Problems In Organizing Information Collection . * 

1, The schedules of a program^s participants, e,g,, employers and 
teachers, do not always coincide with the evaluation schedule. 

2. The most desired information collection techniques may not be 
the most reliable or valid, and they may be the most expensive. 

' 3.. The evaluator might prefer a particular design,s but be unable to 
Implement It because comparison groups are unavailable, students may move, 
.X data cannot be Identified from previous years, etc, 

k. Difficulties arise In obtaining Information about eligible par- 
ticipants becasue of privacy regulations, inability to obtain participant 
cooperation, and mobility. 

5, Career preference tests and Interest inventories may be biased, 

thus, limiting the job options available to women and ethnic minorities. 

y . 

A Case Example 

For an evaluation of a cooperative education project at Webster City 
College, USA, the evaluation questions focused on the requirements of P.L. 
SkrMSl. Two were: 

1. Was there a difference between men and women, with respect 

to their ability to find employment In occupations related to their training? 

2. Did employers changes their views about students,* ability 
and^ training from the beginning to the end of the program? 

To answer these questions, the evaluator had to select rcillable and 
valid Information collection techniques, choose an evaluation design, and 
select a sample that was sufficiently large enough to produce all the 
needed Inf ormat Ion 

For the first question, the evaluator decided to use face-to-face in- 
terviews with some students and to send questionnaires to the rest. From 
the start, the evaluator was made sensitive to the possibility that some 
of the women might be angry about the relationship between their training 
and employment because they perctelved their employment opportunities to 
be more limited than the men's. To ensure the precision of the findings, 
the evaluator planned to describe and clarify the extentof these feelings 
In the evaluation report. ^Further, the evaluator used a comparison group 
group design that can be Illustrated as follows: 
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Men 

Ability to Firfd 
Work Related to 
Training 



vs Women 

Ability to Find 
Work Related to 
Training 



All students In the program were Involved in the evaluation. 

For the second question,- the evaluator used telephone interviews as 
the major Infqrmation collection technique. The evaluation design em- 
ployed was a time series that can be Illustrated as follows: 



Beginning 
of Program 


Employer's Views 


End" 


Employer's Views 



All employers were Included in the evaluation. 
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ill COLLECTING INFORMATION 

Collecting evaluation information is a large and complex task that 
has a diraet bearing on' the quality of the resulting evaluation Informa- 
tion. Poor information collection instruments can yield Invalid and pos- 
sibly false Information. 

Considerations ■ Involved In Successful Informatlo.. Collection 

1. Selecting, adapting, or developing instruments: The first st6p 
In collecting information for the evaluation of cooperative education pro- 
grams Involves the evaluator in selecting, adapting, or developing reli- 
able and valid instruments to measure the effects of the program. 

2. Hiring and training information collectors: Information collec- 
tors can be selected from the program staff Itself, professional organiza- 
tions, -and the community. Once hired', collectors must undergo rigorous 
training. 

3. Pilot testing information collection Instruments and procedures: 
Before using information collection instruments and procedures, they should 
be pilot tested to help answer qQest Ions like: 

• How accurate Is the information obtained with the instruments 
(validity)?' 

• How consistent is the information obtained with the instruments 
(validity)? 

4. Conducting information collection: Information collection can 
mean obtaining "clearance" from agencies like the U.S. Office of Manage- 
ment and Budaet (OMB) and informing participants of the purpose and nature 
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of their cooperation. 

5. Monitoring Information collection: Information collection should 
be supervised to ensure that activities are being conducted correctly, and 
that all necessary data is being gathered. 

Problems in Information Collection 

1. Difficulties arise in identifying val idatedMnstruments , and 
development is expensive or time-consuming. 

2. The best Information collectors are unavailable or too expensive 
to hire or train. 

3. A pilot test can be too small or inadequately performed to provide 
reliable Information. 

k. It Is sometimes difficult to allot the necessary amount of time 
^ for clearance (often several months) of newly-deve loped Ihstruments, and 
validated ones are not available. 

5. Once informed, eligible participants may withdraw from the evalua- 
tion. 

6. Difficulties arise in obtaining cooperation of participants, 

7. Difficulties arise in collecting information frotn participants 
who move away or lose interest in the program after completing their for- 
mal participation in it. 

A Case Example 

Salem Community College is known for its excellent cooperative edu- 
cation program.. It has a reputation for providing outstanding training 
in many fields, and is particularly proud of its Computer Technology As- 
sistance Program (CTAP) . For an evaluation of the productivity and im- 
pact of the first three years of CTAP the evaluator will be conducting in- 
terviews, sending out questionnaires, making observations, and administer- 
ing achievement tests and attitude surveys. Because of the large number of 
different information collection techniques being used, the evaluator , at- 
tempted to find instruments that were used in other similar evaluations. 
The purpose of the search was to save the time it takes to create an in- 
striiment. Unfortunately, only one achievement test and one observation 
form were found to be adequate. During tha search, the eva'luator disco- 
vered that at least one of the available attitude survey instruments was 
sex-biased in that it assumed that a repair of a keypunch machine or com- 
* piiter console was necessarily made by a male. 

Having selected the information collection instruments, the evaluar.or 
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then submitted them to the Office of Management and Budget for clearance. 
Once the Instruments were approved, a pilot test was conducted to ascer- 
tain their rellaljlllty and validlr.y. 

Information collection took fourmonths. The evaluator carefully su- 
pervised the process, and'Jn so doing, discovered that the people con- 
ducting observations were not uniformly using the form prepared for them. 
Thus, all observers were retrained In order to ensure that Information col- 
lection was* proceeding as snxjothly and accurately as possible. 'Another 
problem was not so easily solved. In the three years since the start of 
the program, at least 20% of the participating students had moved, and no 
forwarding addresses could be obtained. The loss of this group was described 
In the final report, and the evaluator noted that there was no way of know- 
ing whether these students were different from the rest In some Important , 
way and how their loss altered the accuracy of the evaluation conclusions. 

IV PLANNING ANO IMPLEMENTING DATA ANALYSIS 

The analysis of evaluation Information Is the process by which all 
the data obtai^ned during the various information collection activities 
are summarized and synthesized to produce answers to evaluation questions. 
Analysis methods range from the statistics-based techniques used by psycho- 
logists and sociologists, to the scholarship-based techniques often used 
by historians and anthropologists. 7\11 attempt to describe evaluation in- 
formation in the form of tallies or measures of variation, and to explain 
evaluation Information by identifying patterns and trends In events. 

Considerations Involved In .Successful ly Planning and Implementing 
Data Analysis 

1. Planning the data analysis: Analysis activities must be care- 
fully planned to be technically appropriate, responsive to the evaluation 
questions, and In turn compatible with the design strategy and information 
collection techniques. The selection of specific analysis methods will 
usually be influenced by the evaluator's training and background and the 
resources available for the eval uation. 

2. Major analysis techniques include: 

• descriptive statistics 

• correlation 

• regression 



• analysis of variance 

• chl-square ^ * • 

Descriptive statistics include averages, frequencies and measures of 
variation like the standard deviation. Correlations describe relationships 
between two variables, while regression techniques are used to examine 
the relationship between a criterion (dependent) variable and two dr more 
predictors (Independent) variables. Analysis of variance techniques are 
used to compare two or more groups in terms of a single variable such as 
achievement. The chl-square statist ic can be used to compare two or more 
groups In terms of a dependent variable by testing whether It is plausible 
that the two sets of empirical data are random samples from the same popu- 
lation. 

3. Conducting data ana,lysis activit : Completion of analysis ac- 
tivities must include more than just the u ual performance of the analy- 
sis; It must also involve: 

• reducing the evaluation information to usable form 

• pilot testing the information analysis accivities 

• conducting the analyses 

• filing the evaluatlc^n information. 

Problems in Planning and Implementing Data Analysts ^ - 

1. The evaluation questions are unclear, and it is difficult to 

< 

tailor the analyses to them. 

2. The eva1uator*s personal training or background influences him 
or her to accept an expensive or otherwise inappropriate analysis method. 

3. The design strategy has been improperly selected or poorly im- 
plemeuted, or the information collectibn techniques are unreliable or in- 
valid, yielding uninterpretable results. 

Too much information is collected, or it Is badly reduced, delaying 
the performance of the analyst's tasks. 

5. Pilot .testing can reveal the need to revise some or all analysis 
techrrlques because thej wrong data was collected. It is insufficient to pro- 
vide the answers to the evaluation questions, etc. ri- 

A Case Example ^ 

The evaluation of Federal Union's cooperative education program in- 
cluded the following questions: 
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,1. Was there a difference between students who participated 
In the program and those who did not, with respect to their performance 
on the Work Sitills Achievement Test? 

2. On the 100- item Work Skills Achievement Test, how many stu- 
dents achieved scores of 1-20? 21-40?. 4l-60? 6l-80? 81-100? 

To answer the first question, the evaluator conducted an analysis 
of variance (t-test) to determine if there was a statistically signifi- 
cant difference in the performance of participants and non-participants. 
For the second question,' the evaluator prepared a tally of the number of 
students obtaining scores within each of the five categories. At the con- 
clusion of the analysis, the evaluation data was stored for future refer- 
ence. 

V REPORTING INFORMATION 

An evaluation report consists of the answers to some or all of the 
evaluation questions and an explanation of the procedures used to derive 
the answers. The evaluation report, whether written or oral,, informal or 
formal, is an official record of the. evaluation. It is through the report 
that the evaluator makes public the activities and findings. Thus, 
it Is essential that the evaluation audience be given easy access to 
reports and that they be clearly written. 

Considerations Involved in Successfu lly Reporting lnformatjo_n ■ 
To be credible, the evaluation report must be easily understood by 
all its readers or listeners, including faculty, students, employers, 
administrators, and funding agencies, in preparing the report, the eval- 
uator should consider including the following: 

• an introduction to the evaluation including its background, 
the evaluation questions, and 1 imitations on the scope of 
the evaluation; 

• the collection of evaluation information. inclQding the 
design, sampling, information collection techniques . and limi- 
tations of the information collection activities; 

• the methods used to analyze the data and their limitations; 

• the evaluation findings, including answers to each evaluation 
question, interpretations, recommendations, and limitations 
on the findings; 

• management concerns, like schedules and staff assignments. 
The Importance of each of these considerations will depend upon the 
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nature of the program, the evaluation, and the purposes for which the 
evaluation report will be used. 

Problems in Reporting Information 

1. Technical matters are sometimes difficult to translate into 
tcr^Tis that all appropriate audiences (e.g., students, employers, and 
funding agencies) can understand without oversimplifying. 

2. It is hard to assign priorities to the infonnation so that 
only the most important is emphasized. 

3. Reconstructing evaluation events can be difficult and time- 
consuming. 

k. Evaluation reports sometimes appear overly critical or too full 
of praise, rather than providing a balanced view. 

A Case Example 

The final report of the evaluation of American Technical Institute's 
cooperative education program was submitted for review. The table of 
contents was: 

I. Introduction to American Technical Institute 

II. Evaluation: An Overview 

A. Evaluation: A Definition 

B. Improvement and Effectiveness Evaluations of Coopera- 
tive Education ^* 

C. Evaluation Questions 

III. Collecting Evaluation Information 

A. Information Collection Techniques 

B. Limitations on the- Collection of Information 

IV. Data Analysis 

A. Methods 

B. Limitations 

C. Results 

V. Evaluation Findings 

A. Answers to Evaluation Questions 

B. Interpretations 

C. Recommendations 

D. Limitations 

VI. The Evaluation Staff 

A. Personnel 

B. Calendar 

The reviewers noted that the evaluator had omitted describing several 
important components of the evaluation and recommended that they be added, 
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Limitations on the scope of the evaluation were not included, the 
reviewers pointed out. Q.uestlons of cost and of effectiveness of the 
program over the five years of Its existence were not addressed, for 
example, because the evaluation funds were simply not adequate to answer 
them accurately.. But a reader without this knowledge might suspect that 
the evaluation was negligent, whereas, in fact. It was prudent. ^ 

Other omissions from the report Included descriptions of the evalua- 
tion-design and sampling procedures. The reviewers suggested the in- 
clusion of these descriptions. They also recommended that the section on 
information collection include an explanation of the strategies used to ^ 
establish the reliability and validity of each Instrument. 

VI MANAGING EVALUATIQH ACTIVITIES 

Ability to manage and coordinate evaluation activities is essential, > 
and at least some portion of the evalua tor's time must be given to manage 
ment. It is only through careful attention to schedules, tasks, and bud- 
geting that the evaluator can assure teachers, students, employers, funding 
agencies, and advisory councils that they will get timely and usable an- 
swers to evaluation questions. 

Considerations Involved in Successfully Managing an Evaluation 

1. Establishing schedules: Evaluations are commissioned to be con- 
ducted within a given amount of time. To ensure the success of the ef- 
fort, the evaluator must determine when each. evaluation activity will take 
place^ the sequence of the activities, and how long each will take. 

2. Assigning staff to activities: In order to assign staff to spe- 
cific evaluation activities, the ski 1 Is'needed to perform each activity 
must be identified so that the staff members with those skills can be 

assigned appropriately. 

3. Budgeting: To prepare an evaluation budget, the evaluator 
must weigh what needs to be done against the amount of money t;hat Is 
likely to be available. Invariably, activities, time allocations, and 
staff assignments are *mod if ied during the development of the budget. 

Problems Involved in Managing an Evaluation 

1. There is never enough time or money to do the perfect evaluation! 

2. Trained staff may be difficult to find. 



A Case Example 

A nine-month evaluation of a new cooperative education program to 
train people for the health professions was commissioned. The cost 
allotted to the evaluation was $20,000. The evaluator worked out the 
following budget and schedule. 



EVALUATION BUDGET 



Direct Costs 

A. Salary and Wages 

1. Donald Smith, Ed.D. , 

Evaluator^ 20% time for 12 mos. @ $2,000 per month 

2. Roberta Clark, M.Ed. 

Research Assistant: 50% time for 12 mos. g $1,000 

per month 

3. Joan Thompson 

Secretary: 20% time for 12 ijios. g $800 per month 

4. Fringe benef i ts(socIal security, iiealth insurance) 
8 20% of salary and wages ($12,720) 

B. Supplies: paper, typewriter ribbons, cassettes 

C. Printi/ig and Reproduction (questionnaires, reports) 

D. Computer Costs (data analysis) 

E. Telephone and mail @ $10 per month ^ 



Subtotal 



II 



Indirect Costs: 20% of Direct Costs 



$i|,800 

'6,000 

1,920 

2,544 
300 
400 
250 
120 

$16,334 

3,^66 



TOTAL 



r EVALUATION SCHEDULE 



I 

vn 
I 



1. Select Evaluation 
Questions 

1. Organize Information 
Col lection 

3. Collect Information 



h. Conduct Data Analysis 



5. Report Information 



Sept. 
1 + 30 



Oct* 
1 + 31 




Nov. 
1 + 30 



Dec. 
1 + 31 



Jan. 
1 + 31 



Feb. 
1 + 28 



Mar. 
1 31 




Apr. 
1 30 



Hay 
1 31 
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CHAPTER IV 
COUNSELING CHECKLISTS 

In Chapters IV, V, and VI, three checklists are provided: 

• Counsel ing 

• Program Acti\Aities^ 

• Performance and Cost 

Each checklist has been designed so that the cooperative education 
administrator can evaluate the reliability, validity, and efficiency of 
existing instruments designed to appraise the impact and productivity of 
their programs or to develop appropriate evaluation designs and techniques 
In the respective areas. The topics for each checklist were derived from 
Information obtained from interviews with program directors, all of whom 
rated each as being extremely important to the evaluation of post-secondary 
cooperative education efforts. 

The checklists have deliberately been prepared to be brief and easy 
to use, while at the same time to be as comprehens ive as possible. The 
reader is referred back to Exhibit 1-1 in Chapter I for a review of the 
several evaluation checkpoints for which the three evaluation checklists 
have been developed in the next two Chapters. 

$ince the checklists are also flexible, no one program would make use 
of every item on all checklists. The program administrator should use 
only what is applicable to his or her own program. There are, naturally, 
space, time, and budget considerations involved in implementing the 
suggestions contained in the checklists, as they present characteristics of idea 
evaluation efforts, i.e., goals to be worked towards gradually. 

I COUNSELING FUNCTION 

Because of the critical nature of the counseling function in post- 
secondary cooperative education programs and its relationship to other 
evaluation checkpoints (e.g., job matching, diagnosis, etc.) a rather 
detailed and comprehensive checklist has been developed, with discussion 
of major issues preceding it. 
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li OVERVIEW AND ISSUES 

The way a cooperative program Is both structured and managed may 
make the difference between a program which matches students with^ jobs 
based on the student ^s background and one which matches students with 
Jobs simply because a job is open and the student needs money. If a 
student has an interest in an area the student will perform better on-the- 
job, leading to greater employer satisfaction and. hence, employer re- 
enlistment in the program. If a student is pleased with the job situation, • 
he or she will probably do better in courses and vice versa * 

1. Who directs or manages the cooperative program? In some colleges 
the cooperative program is managed by the financial aid staff (National 
Association of Student Financial Aid Offices, 1975). Several problems' arise 
from this situation. The financial aid officer is not trained in helping 
students decide on careej paths, linking interests and skills with jobs, etc. 

The instructional faculty could also structure and manage the coopera- 
tive program. However, the faculty rarely has the staff or time necessary 
to handle tjie details and follow-up required for this program's success. 
In addition, they may not have the skills needed to help students decide 
on career paths nor help them match job skills with coop jobs. 

The placement officer might structure and manage the program. If this 
occurs, it is imperative that there be an active follow-through program to 
help students during the cooperative program. The placement officer may 
not have the staff to do this. Unless the cooperative program had a high 
priority in the placement office, cooperative students might not get the 
requisite attention or maximum benefit of a cooperative education experience. 
The placement officer might be reluctant to handle problems with an em- 
ployer since this might limit future jobs or recruiting for permanent em- 
ployees. The evaluation of the placement office may be predicated on the 
number of students placed in jobs rather than how many students are placed 
in jobs that relate to their career goals. 

The cooperative program could be managed and directed by the student 
employment office. After all, it is a paid work experience. However, it 
''is more than this. It is a learning-growing experience. If a student is 
seeking financial aid only, then the student employment office might be of 
help. However, the cooperative education program's purpose is to provide 
an opportunity for the student tj^^ink a job experience with a learning 



experience to use as a framework for testing or preparing for a job* 
Most student employment experiences cend to be dead-end jobs without 
provision for upgrading or upward mobility. 

The cooperative program could be managed or directed by the counseling 
officer* This would provide the student with an opportunity for self- 
exploration and career exploration. However, if the cooperative program 
were the responsibility of the counseling department, it would be "impera- 
tlve that the counselor or counselors working with the students have training 
In vocational counseling. Many counselors are trained in a therapy milieu 
and do not have experience in vocational issues (Hilliard, 1977)»- 

One solutijn would be to have a special cooperative staff with Input 
from the financial aid office, instructional faculty, placement office, 
student employment office, and a strong counseling component. This might 
be the best of all worlds but only if a strong liaison among all these ele- 
ments IS or can be a reality without the problem of **turf fighting". 

2. Who generates cooperative jobs? The generation of cooperative 
jobs is a critical ingredient in the cooperative program. Cooperative 
jobs may be developed in many ways. However, since a job wi th^ potential 
to be upwardly mobile is required, the job developer should understand the 
job market, employers* objectives, and types of jobs that will provide 
students with a growing experience. Often students are left to fend for 
themselves and told that if they are interested in participating in the 
cooperative program, they should^ find their own jobs(Nat ional Association 
of Student Financial Aid Offices, 1975)- Students may not have the re- 
sources or background knowledge with which to do this. 

A professional job developer may be an ideal solution. This person 
should be knowledgeable in current employment trends and be able to work 
easily with employers. Other offices such as the placement office, the 
student employment office, or former cooperative students could provide 
leads to jobs. Student employees might serve as interns to help locate 
jobs. Generally, they probably would not have the background expertise 
to handle the details of job development. However, job developers might 
be paid staff hired by the student employment office. The entrepreneurs 
among these students might want to work on a commission basis. 

3. Who evaluates cooperative jobs in terms of skills and job 
relatedness? Evaluation of the cooperative job in terms of skills and 
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Job relatedness is a sophisticated task. More will be said later about 
matclilng course work with job skills. Yet, It Is essential in terms of 
running an effective cooperative program where skills are related to jobs. 
For this task, a person trained in .vocational development Is needed. 

If. How do students learn about cooperative programs? Too often 
students blunder Into the cooperative program simply because they can 
not get financial aid elsewhere or think that it might be Interestijig. 
No one knows how many more students might enter the" cooperatiA/e-program 
If they knsw about It and it was given a high priority at t!ie school. 
Certain schools do emphasize the cooperative program and have large seg- 
ments of their students enrolled in a cooperative program. 

Students should be exposed to all of their options for financial aid. 
This might be handled through materials mailed to students prior to a 
students entering the College, Interviews with perspective students in 
the financial aid office, and/or as a part of orientation to the school. 
However, students should be aware that the cooperative program Is not 
just an optional way to make money, but a means to explore jobs or careers. 
In this framework, before students enter the cooperative program, they 
should talk with a counselor about their vocational goals. Generally, it 
is the counselor who has the experience to help the student In this area. 
Most of the people in the other choices listed do not have the background 
or expertise to help a student fully understand the steps in self-explora- 
tion, career exploration and career development. 

5. How does a student match her or his vocational de velopment with 
cooperative Jobs? When there is no one to advise a student about the jobs 
offered in the cooperative program, the match occurs by chance. Many stu- 
dents may not even be familiar with the range of skills they may need in 
various careers and how one job experience may be similar to another with 
different titles or what seems different on t'le surface (National Associa- 
tion of Educational Progress, 1976). 

There are several systems to help students make a match between their 
Interests and jobs available, special tests of interest, computer matching 
programs, etc., but it is essential that the student have an opportunity 
to discuss the results of interest inventory tests or the computer printout 
on her or his individual form with a person who can bridge the gap between 
the individual and what is offered (Stebbins, 1976). This is a critical step. 

45 
-39- 



Talking with employers may help, but too often the employer does not 
have the time to present the options to studnets. Friends or family may 
provide some help, but with disadvantaged students in particular, this 
option does not usually provide much positive information. The placement 
director probably does not have the time to help the student come to clear- 
er decisions about some hazy career potential. As a conclusion,"^ it is 
generally the vocational ly- trained counselor who could provide the help. 

6. If students were not able to secure cooperative jobs, to whom 
can they turn for help? The answer depends on why the students did not 
get jobs. If the students needed the cooperative jobs solely for finan- 
cial support, they should- be referred to the financial aid officer, who 
could possibly package another type of ^lan for them. If the students 
were not able to secure a job bjecause there were no jobs In a field of 
interest to them, then the counselor or advisor should be able to suggest 
other options including summer employment, internship following graduation, 
voluntC:er work, etc. Perhaps other courses could help the student become 
more employable. The student might have to travel outside of the area. 
The exploration of options such as these may best be handled by a person 
with experience in job variables. A student may be hindered from paining 

a position because of a personal problem or self-defeating behavior. It 
would be the job of a counselor to help the student explore what the pro- 
blem is and work out ways to help the student overcome this behavior or 
refer the student to other counselors or agencies for help. 

7. To whom can students turn for help during the program? When stu- 
dents encounter roadblocks, they need someone to talk with and help them 
find solutions to their problems. On some occasions the problem may be 
employers' unrealistic demands or an employer's attitude. In these cases, 
the person running the cooperative program should talk with the employer 
or bring the employer and emp' >yee together in a neutral atmosphere to 
air the problem and help them work out a solution. Perhaps it is the em- 
ployee's attitude or self-defeating behavior, as discussed above, that 

Is the problem. In this case, the person running the program should ei- 
ther counsel the student or call upon a counselor or outside agency for 
help. 

8. To whom can students turn for help in evaluation of their 
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experience? Once students have completed the cooperative program, they 
should have the opportunity to evaluate their experience in terms of their 
personal and vocational goals. Perhaps the experience taught the student 
that the ^rea he or she worked in was not suitable. In this case, another 
round of career exploration might be necessary, or Smother type of em- 
ployer might lead the student to a different conclusion. 

Perhaps the student has time to re-enter the. cooperat h^e program,' 
so the cooperative director could help the student find another coopera- 
tive job that might provide a more positive experience to advance the 
career goals the student has reformulated. If there is not time, the 
student should be directed to a potential employer, or at least be 
given directions on what the next options might be. Assertive students 
might profit from reading and doing the exercises in books such as Go Hire 
Yourself an Employer (! rish, 1973)- 

Evaluation of the program for the student is as Important as evalua- 
tion of the cooperative program by the school. Without evaluation, stu- 
dents lose their way and may not understand how their experiences contri- 
buted to their goals. 

9. Where can students get additional up-to-date information on 
other aspects of this career or related careers? The school should offer 
an up-to-date career library. As long as the student has access to the 
\material, 'the location of the library may be unimportant. To ensure max- 
imum access, various libraries may be located on a large campus. Instruc- 
tional staff should be encouraged to obtain such material and put it into 
a central display or departmental display that is available to the student 
located, perhaps, in the s ' library or counseling center. 

Many career libraries are woefully out of date. This presents several 
problems: much of the older material is sex biased; the salary or wages 
suggested may be out of date; new equipment or opportunities might not 
even be mentioned. A member of the cooperative staff should have the 
specific duty of working with the regular counseling staff to ensure that 
the career material is current and non-biased. 

Use of a computer terminal may help the student gain access to cur- 
rent career information. The state employment office also should have 
figures by the state or region on employment. However, it may not be in 
the most useable form. 

10. Who keeps records on success of the cooperative program? In a 



school with high personnel turnover, it is important that the record- 
keeping system be independent of any one ^individual but the; responsibi- 
lity of a designated category of persons. Too many programs flounder 
because of inadequate records. It is important, however, that adequate 
secretarial or clerical staff be -available to free the professional 
cooperative staff of the burden of paperworlc. To expedite the paper- 
work, forms should be designed to be efficient and not duplicative. \ 
Whoever keeps records on the program should understand the Students 
Right to Privacy Amendment. Thus, records should be the Vesponsibil ity 
of a professional staff. Cooperative records should be kept intone 
pvace» ' \ 

11 . A breakdown of all the following records by sex, r^ce, eth- 
nicity, handicap, and age group is necessary. It is important that 
records on the success of the cooperative program be kept by the variables 
of sex, race, ethnicity, handicapped, and by age groups. Examination of 
such records will permit a school to determine if they are In compliance 
with the Vocational Education Act of 197^, Title IX of the Education f 
Amendments of 1972, and civil rights statutes on race and the handicapped. 

Status records kept by broad age-groups will help the school deter- 
mine if those people experiencing mid-career changes^ or women returniing 
to the work force, are placed in cooperative programs as regularly as | 
younger students. It would be important for a school to assess how well 
it functions for these groups. 

Examination of the results of such records will permit a school to 
see how well it is meeting the needs of its constituents. It may be \ 
found that fewer women drop out of cooperative programs if they receive 
counseling that helps them cope with triple career stituations of being 
mothers, students, and employees (Wirtz , 197^; and Stebbins, 1976). Per- 
haps handicapped students who received counseling were more readily ac- 
r^jpted by employers than those who did not. The results of this record- 
keeping may be instructive for the re-direction of the entire cooperative 
education counseling program. 
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Necessary Records ; 

(a) Percentage of placement In cooperative jobs after counseling 
compared to percentage of placement without counseling . 
Assuredly, numbers alone may not be the answer since 100 
students placed in cooperative jobs may have 100 mismatched, 
unrewarding experiences based simply on taking the first job^ 
without counseling. Again, if there are Insufficient numbers 
of cooperative job options open to students, this figure 

and counseling for the few jobs may be Irrelevant. However, 
the numbers may be the beginning of the story. The evaluation 
of the program shouf^ look at the numbers jn an attempt to 
discover how they were generated. 

(b) Percentage of attrition in cooperative program after si^ 

^ months for those who received counseling compared to those 
who did not . According to Karabel , only 38? of the students 

\ enrolled In community colleges complete the course in four 
years (Karabel, 1972). The percentage of attrition froi-n 
Cfooperative programs is not known (Abramowltz, 1977)* 

(c) Satisfaction of students with cooperative Job match when 
they received coun sel ing compared with students who did not . 
Since satisfaction is a subjective term. It Is difficult to 
measure on an absolute scale. However, some of the numbers 
generated by the previous question may provide the answer. 
Simply asking students the following questions may provide 
another measurement: "Are you satisfied with the match be- 
tween your career plans and your cooperative job? Did 
counseling help you in this match?,** and the addition of the 
question,; "WhVt did cc*:nseling do to help you?,*' might pro- 
vide some clue\to thDse elements of the counseling program 
that are most successful. 

(d) Satisfaction of employer v/lth cooperative job match of those 
students who received counseling compared .with those who did 
not . Match the employers* response to a simple question, 
'*Are you happy with the work and attitude of your cooperative 
studerjt?'* into two categories: those students who received 
counseling and those who did not. An examination of the two 
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categories should provide an answer, 
(c) Salary received by former cooperative students compared to 

those In similar jobs who did not, participate in cooperative . 
programs . 

(f ) Percentage of cooperative students who hold jobs in their 
area of interest after college compared with percentage of 
non-cooperative students who hold jobs in their area of interest 
after college . The placement office, in their follow-U|j, 
should gather this material and then provide it to the co- 
operative program director, who could give it to the counseling 
department for analysis. 

Iir COUNSELOR- STUDENT RELATIONSHIP 

1 . TKe counselor's attitude reflects respect for the student . 
Basic to any effective counseling program is respect between the counselor 
and the person counseled. This is not a simple matter but may be achieved 
by a white counselor working with minorities to help minority students 
overcome hostility or disbelief that a white counselor will be able to 
address their problems. 

A similar caution is also important for those men counseling women. 
Frequently, women returning to school may be unable to articulate their 
needs in traditional terms but need to be encouraged and have someone 
help them translate their skills and abilities into job categories. 

2. Ways are provided to help students explore their self-iden- 
tity and personal goals . Often students have very unrealistic or con- 
fused attitudes about themselves. Many have never thought coherently 
about their possible strengths and weaknesses. If a student is able to 
do this without much difficulty, then a book such as Where Do I Go From 
Here With My Life (Crystal, 1976), or a similar one, might be useful. 

However, if a student has identity problems or fears about achieve- 
ment (Healy, 197^), then such an approach may be limited, and a more 
guided, structured approach should be used. This might Include several 
of the, tests or computer matching programs on the market. It might also 
Include group counseling sessions, perhaps including parents or spouse 
(Stebbins, 1976). 

3 . Ways are provided to help students learn about car ee r clus- 

i. 

ters . The idea of career clustering Is not a difficult one. It can be 
explained through a large poster on the bulletin board, or simple memeo- 
graphed handouts. The advantage of this approach, regardless of which 
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set of career cluster approaches is adopted, Is that it provides a struc- 
ture for the student in choosing an approach. Many convnercial products 
now on the market apply an occupational or career cluster approach • 

k. Counselor explains ladder approach to careers and the'ten^ 
dency toward frequency of job change and job mobility. The trend in edu- 
cation and careers is less rigid than in previous times. Many students 
begin college, drop out, and later return to college. Most careers are 
structured so that a person with lit:le experience begins at the bottom 
and through more experience or education advances vertically in a career. 
However, in a tight labor market there is mere horizontal advancement. 
When a person feels hemmed in by a job, it^ :3 time to consider getting 
a new job. Students may not be familiar with building up skills and 
contacts on a number of jobs, then moving on to other jobs that require 
more skills. The concept of vertical and horizontal mobility should be 
explained to students. 

'ihe average number of jobs each person holds during his or her life- 
tlrre is five, with the number increasing. Several individuals have com- 
plete career changes as well as job changes. 

5. Information is provided on various lifestyles and their con- 
sequence s . Many careers involve differer.t lifestyles. Students should 
have knowledge of the consequences of a certain career when making voc^ 
tional choices. 

6. Counselor provides some information on all career poten- 
tialities t o both men and women and people of minority groups. Counselors 
may need training in sex-fair and race-fair concepts to understand what 
this means. In the past, it was thought that certain jobs were only ap- 
propriate for one sex or race (Stebbins, 1976). 

A three-page checklist on implementing a sex-fair career guidance 
program is in Sex Fairness in Career Guidance; A Learning Kit (Stebbins, 1 

Many women and minorities may be reluctant to think about non-tradi- 
tional careers because of prior socializing. Women need to see role models 
in career material. Women and minor! fi^^.s may need additional counseling 
to help them break their own ant i -achievement atititudinal barriers prior 
to discussion of careers (Irish, 1973; and Crystal, 197^). 

7. Counselor has current information on various careers and has 
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evaluated {t for sex- and race-fairness. Examination of career material, 
be it comtiercially prepared or industrially supplied, should be instruc- 
tive. How many women are pictured in non-traditional jobs? How many mi- 
norities are in these positions? is the masculine pronoun used exclusively? 
Much current career material is more sex-fair than that of ten years ago 
(Burnett, 1975).^ Currenty information is also necessary because many 
entry-ievel or technical jobs have emerged or shifted considerably in the 
las% decade. 

A useful list of career materials is in Sex Fairness in Career Guidance 
(Stebbins, 1976). However, counselors should supplement the material by 
contracting industries or groups participating in the cooperative program 
to get career information. Other information sources include local, state, 
or national industries, their trade associations, or unions. 

8. Counselors help students evaluate cooperative jobs in terms 
of skills required. The employer should supply a description of the job 
which is broken down into skills. A job description which is not described 
in skill terms should be analyzed by instructional faculty or persons know- 
ledgeable in the area to break it down. Once this is done, if a student 
has., a choice among several jobs, it would be easier to make a decision on 
which job would offer the student more opportunity in an area that he or 
she is seeking. 

. 9. Counselor works with students in developing job a ttack skills. 
Students may be turned down for a cooperative job if they do not present a 
favorable image to the employer. The counselor should help students or 
provide paraprofessional help for students in resume writing, interviewing, 
follow-up, selecting references, etc. (Irish, 1973; and Crystal and Bolles, 
197A). 

Many counselors see this step as beneath their professional duties. 
However, it often makes the difference between a student obtaining a job 
or being unemployed. Such skills might be taught in a required short- 
term course for cooperative education students. 

10. Counselor helps students plan course(s) to take that would 
help fulfill vocational objectives and cooperative requirements. Generally, 
a student has a choice of several paths. The counselor should be able to 
show students the consequences of each course; i.e., to complete a graphic 
arts major, a student could take a course in fine arts or technical draw- 
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Ing. The counselor could help the student see which choice would help 
make him or her more employable in terms of the types of cooperative 
Jobs open and the skills employers require. This may be one of the 
cr-l t lea r areas of decision for the student, hence theneed for the coun- 
selor to be wel 1- informed about the content of the courses offered and 
the relationship between the courses and the world of work. 

1U Counselor acts as ombudsperson for the student with the 
adminlstretion, faculty, and employer. 'Tr.a role of the counselor as- an 
ombudsperson Is a vital one. The counselor should intercede on behalf^ 
of the student to allow the student to register at a different time l^ 
the one time allowed cannot be met by the student, to allow the studei|t 
to take a needed course out of sequence or when the course is over-sub- 
scribed, etc. Frequently students become lost In the mechanization^ of 
the computer world of the college and feel powerless to take steps 'toi 

stralghteni things out. The counselor could either help directly or 

<^ - ^ 

oversee and direct the student in unravelling the confusion. - 

A student may need special help from a faculty member but be afraid 
to ack; the counselor as ombudsperson might Intercede here, too. 

If an employer makes unrealistic demands upon a student, the student 
may be pr^erless to refuse because a grade hinges on the employer's evalua- 
tion. Here too, the counselor should step in and not provoke a situation 
but, acting an arbitrator with a bias toward the student, the counselor 
should try to help the employer and student come to a resolution. A model 
for doing this might be adapted from the role of mediator In integrating 
a school (Lincoln, 1976). 

12. Counselor helps students with problems as they arise during 
the job. For many students, this cooperative job may be their first job 
experience. As such, they may not understand proper behavior on the job, 
(fress, attendance, or need to be on time. Such problems might be ironed 
out better In a course prior to the cooperative job, but when they are 
not or when Ihey re-appear, the counselor should have sufficient contact 
with the empiloyed students to call them in for a chat to help them suc- 
ceed better.; If the job demands are more skilled th:sn the student can 
handle, the /counselor might go to a faculty member in the student's field 
and discuss the problem to see if the job could be broken down into simpler 
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components or if someone could tutor the student to help him or her do 
the job- 

13, Counselor helps students evaluate their experience after 
a cooperative program^has ended in terms of permanent employment options 
and preferences. Once the cooperative program has ended, evaluation of 
the experience is essential, in many cases, the student receives a grade 
or credit for the experience but there is not discussion of what the ex- 
p'erience taught the student. Perhaps, evaluation might reveal that the 
experience was atypical and other jobs in this area might be different, 
or the student did not enjoy the work and the result of the cooperative 
experience was learning that he or she did not want to work in that area. 
At that stage, it is important for the student to begin the career evalu- 
ation steps once again. The student may have an opportunity for another 
cooperative experience, or the student may decide to switch fields of 
study. The counselor should be able to provide a framework to help guide 
the student in a post-job process of sorting out thoughts .and experiences 
to build toward the future. 

Then the counselor should, either by working closely with the place- 
ment office or by functioning as a knowledgeable job market expert, be 
able to tell the student about the opportunities in various fields in the 
local area, state, and nation. Perhaps such a post-cooperative experience 
discussion will help students find a relevant job. 

IV COUNS ELOR - ADM I N I STRAT I ON REIAT I ONSH I P 

Counselors are in a position to see scheduling conflicts. For ex- 
ample, a required course in English may only be offered during the spring 
semester, but this is when some students are on the work phase of a coop- 
erative program. The counselor could recommend ways to resolve this di- 
lemna and eliminate it in the future. Registration for courses for the 
•next semester may occur at a time when the student is off campus on the 
cooperative job; hence it may be necessary to let cooperative students 
register by mail or give them preference by early registration. 

Aff irnfiative action is imperative in all aspects of the college but 
it should especiai.ly be so in the cooperative program. The administration 
should be taught that numbers alone in placement do not evaluate a program. 
The numbers should be balanced by a substantial number of minorities, wo- 
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nicn, and handicapped individuals in non^-tradlt^^'onal jobs. Examination of 

records suggested in t^e overview will provide a valuable means for evalu- 
ation. ^ 
The cooperative program* s counseling staff should meet the college's 
affirmative action program; i.e., should be balanced in terms of minor i-, 
ties, women, and handicapped. Counselors should help administrators with 
the paperwork on TTtle IX, the specific recordkeeping required for the 
Vocitlonal Education' Act of 1976, and other civil rights laws as they re- 
late to the cooperative program; Then, these records should be used for 
an Inst i tut ional sel f-eva 1 uat i on. 

Is the caseload of the counseling staff realistic for the job that 
is supposed to be done? Does the professional staff have the necessary 
support staff to handle the paperwork and other duties? Examination in 
this area could result in cost savings and better development of resour- 
ces. Other cost-saving measures might include use of paraprofesslonal s 
or peer-counselors to do routine tasks. However,, It is vital, that these 
people receive training from the counselors and supervision by them. 

It js also Important that cooperative counselors refer students who 
need other types of help to . ner counselors or agencies for it. However, 
students should not feel fragmented by the various types of counseling or 
help they receive, so either the cooperative counselor or another individ- 
ual should have responsibility for the total individual, to bring together 
the various needs the student has into one wholistic set of records and 
in one person who would have an overview of the total person. Counselors 
should keep in touch with what Is happening in other types of counseling 
programs. 

It may not always be cost-effective to counsel students Individually. 
Group counseling is a growing popular option. However, counselors must 
be trained in group counseling since the dynamics of a group are different 
from one-on-one counseling situations. 



V COUNSELOR- FACULTY RELATIONSHIP 

The cooperative education program depends on the course v;ork that 
students take In conjunction with their job experiences. The counselor 
can serve as the link between the two elements: school and job. 

The counselor as ombudsperson , can discuss Individual student's pro- 
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blcms with faculty members. Then, based on feedback from students, the 
counselor can. help instructors modify their courses to make them more 
Job-related. If this is a known part of the counselor's job there would 
be less faculty resentment to it. At the beginning of the year, the coun- 
selor might lead a faculty cliscussion on the cooperative education pro- 
gram and ways to make courses more job-related. The counselor who re- 
ceives feedback from students and employers might, for example, discover that the 
type of printing presses used in the schools are no longer used commercially, 
*t; the students taking printing classes would not necessarily have a 
successful cooperative experience. 

Counselors, might be used as resource specialists to help faculty 
members eliminate racial and sexual prejudices. Faculty in technical 
fields might learn how to encourage women to enter their courses. 

"Faculty don't normally become involved in career advising or coun- 
seling with students because of a lack of motivation, their attitudes 
towards education and work, or insufficient knowledge. However, faculty 
Involvement as student advisers at the University of Northern Iowa 
changed this'' (Routh, 1977). The change resulted from faculty getting 
feedback on unemployment, job satisfaction and campus recruiting, atti- 
tudes of graduates classified by major, identifying whether they expec- 
ted to change jobs next year, and how satisfied they were with their em- 
ployment and average starting salaries. 

"Instead of complaining about faculty indifference, placement offi- 
cials and counselors can develop programs to affect faculty attitudes, 
motivation, and knowledge. Conducting career conferences exclusively for 
faculty, sharing employment and career information with departments, de- 
veloping resource materials classified by major, and arranging visitation 
programs are ju'st a few approaches that help faculty become more involved 
In career planning and placement activities," Routh concluded (Routh, 1977)- 

VI counselqrtEmployer relationship 

The success of the cooperative education program depends on the num- 
ber and variety of jobs available. Part of the job of the counselor 
should be job development. It would work best when the employer gets to 
know and trust the counselor for his or her integrity in sending students 
who can fulfill the needs of the job and not burden the employer with much 
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unnecessary paperwork. Yet the counselor has a responsibility to provide 
sufficient help ar.d background information for^the employer to understand 
his or her responsibilities to the student. This would include informa- 
tion on non-discriminatory hiring, promotion, and ways the evaluation of • 
the student can be conducted* 

It is also the counselor's responsibility to bring the employer Or 
employer's groups, such as the Chamber of Commerce, of business-social 
groups, such as the Jaycees, to campus to meet with the faculty and ex- 
plain the realities of the workplace. The counselor could well serve 
as a vital person on a vocational education steering committee or advi- 
sory council for the region or area. 

Counselors could enlist the goodwill of employers by inviting vhem 
to the campus to meet with students in job affairs.or In individual ses- 
sions prior to recruiting. This building of bridges would probably lead 
to greater satisfaction with the program on the part of the employer. 

Nothing succeeds like success. Employers who felt that the program 
was successful will continue to participate in it and may encourage other 
employers to enter it. When employers drop out of the program in large 
numbers, it is time to re-evaluate the program and see what went wrong. 
It may be possible for the counseling staff to remedy the problem. Pro- 
grams that are successful are well publicized. Encourage the school 
newspaper to carry articles on the cooperative education programs and 
local employers. Have the cooperative education staff meet with commu- 
nity groups to explain the goals of the program and how it is functioning. 

VI I COUNSELOR PREPARATION AND BACKGROUND 

1 , Does counselor hold a degree in vocational counseling, or at 
least one course in it? While a degree is no guarantee of an effective 
counselor, it would provide the person with a framework for understanding 
the steps in helping a person choose an appropriate career and make an ap- 
propriate match between a cooperative job and a person's personality and 
skills. Counselors tend to be trained in individual problem counseling. 
This presents a problem when a counselor is called on to perform vocational 
counseling which some counselors see as a job for a paraprofess ional . If 
a person had a limited vocational counseling background, in-service expe- 
rience and supplemental reading it might help a person gain experience. 
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If a counselor has not been exposed to the new legislation's re- 
quirements regarding sex fairness, race fairness, and handicapped fair- 
ness. It would be wise to contact any of various groups knowledgeable 
In this area to put on a seminar or workshop. A list of participants 
In the Sex Equality and Guidance Opportunities Workshop ^mlght be one 
place to start. Contact with local National Organization for Women might 
produce other ideas. Workshops are also sponsored by various profes- 
sional groups and commercial enterprises. 

2. is counselor a member of professional counseling organization 
concerned with vocational guidance? Such an organization might be Na- 
tional Vocational Guidance Association, or the American Vocational Asso- 
ciation. Through, reading of professional literature, attending workshops, 
and by discussions with other professionals at meetings, a person can be- 
come up-to-date and do a better job. 

Affiliation with such a group might offer the counselor a non-fearful 
method of evaluation by peers, perhaps by observation, role-playing, or 
video-taping of interviews. 

3. Does counselor have any paid work experience other than in educa- 
tion? To work in a cooperative program, a person should understand the 
demands of paid employment, preferably in technological or mechanical occu- 
pations. Many schools require that their vocational teachers have prac- 
tical experience in their field and not simply be educated in the area. 

A similar requirement should be mandated for counselors. 

4. Is counselor familiar with at least four tests of career guidance 
inventories? Does counselor understand differences In their make-up and 
purpose? Simply giving a student a test of career interest is not adequate. 
The right test should be chosen and then fully discussed with the student. 

A counselor who uses only one test for all students may be short- 
changing the students and not fitting the test to the student. A full 
discussion of this problem appears In the Sex Fairness in Career Guidance 
chapter on "Guidelines and Recommendations for Sex-Fair Use of Career 
Interest Inventories" (Stebbins, 1976). 

Career interest inventories are only one source of information use- 
ful for the career selection. Student values, personal i cies , aptitudes, 
achievements, aspirations, family and educational background, are also ex- 
tremely important to decision-making. 
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' 5» Has the counselor experience or training in counseling women 

for nori-traditional jobs? Generally, in order to be receptive to a non- 
traditional job, a woman must have confidence in herself and her ability 
to make decisions. This may require personal or group counseling or asser- 
tlvcness training. Then, a woman must realize how her early socialization 
pro.cess has affected her, how she should not fear a career or earning a 
high salary, and that she can combine a career with being a wife and mo- 
ther (Stebbins, 1976). It is then important for her to read or see occu- 
pational information that is non-sexist, especially that which is sex-fair 
and shows women in non-traditional careers. 

The counselor should feel comfortable about recommending non-tradi- 
tional careers. The counselor has, an additional responsibility to be 
aware of when counseling minority women (Yu, 1976). 

Specific exercises for counselors are discussed In Sex Fairness In 
Career Guidance , in the chapters on "Encourage clients to explore chal- 
lenging, perhaps non-traditional career options*' and ''Explore with clients 
possible conflicts and problems they may encounter when entering a new 
career^area" (Stebbins, 1976). 

6 . ^ ts the counselor aware of Title IX of the Educational Amendments 
of 1972 as it relates to counseling and other aspects of the cooperative 
program? 

"A recipient (of Federal funds — practically all educational agen- 
cies) shall not discriminate against any person on the basis of sex in 
the counseling or guidance of students or applicants for admission. 

"A recipient which uses testing or other materials for appraising or 
counseling students shall not use different materials for students on the 
basis of their sex or use materials which permit or require different 
treatment of students on such basis unless such different materials cover 
the same occupations and interest areas and the use of such different ma- 
terials is shown to be essential to" eliminate sex bias. Recipients shall 
develop and use internal procedures for ensuring that materials do not 
discriminate on the basis of sex. Where the use^ of counseling test or 
other instrument results in a substantially disproportionate number of 
members of one sex in any particular course ot study or classification, 
the recipient shall take such action as is necessary to assure Itself 
that such disproportion Is not the result of discrimination In the In- 
strument or its application. 

"Where a recipient finds that a particular class contains a substan- 
tially disproportionate number of Individuals of one sex, ^ne recipient 
shall take such action as is necessary to assure Itself that such disprb- 
portion is not the result of discrimination on the basis of sex in coun- 
seling or appraisal materials or by counselors." 
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Title IX regulations affect other aspects of the cooperative education 
program, for instance, the need to provide equal Icinds and anwunts of fi- 
nancial assitance. When employment is made available to students through 
•the school Itself or through an outside employer or agency, it must be 
without sexual discrimination. In fact, the school shall not provide 
service to any agency, organization, or person which discriminates on 
the basis of sex in its employment practices. Specific guidelines are al- 
so laid out for prohibition of discrimination on the basis of sex in em- 
ployment, in education programs and activities. This covers participation 
in such activities, application for them through recruitment, advertising, 
hiring, upgrading, promotion, job assignments, classifications and struc- 
tures, granting and return from leaves of absence including pregnancy and 
child care, as well as pre-employment inquiries as to marital status of 
applicant. 

7. Is counselor aware of Title VII of the Civil Rights Act as it re- 
lates to counseling and jobs? The U.S. Equal Employment Opportunity Commis 
sion has issued "Guidelines on Discrimination Because of Sex." Previously 
this law was restricted to prohibition of discrimination because of race 
or ethnicity; now it covers sex, age, and handicap. 

The rules explain the statement that "sex as a bona fide occupational 
qualification" is suspect because it tends to deny employment opportunities 
unnecessarily to one sex or the other. This includes the refusal to hire 
a woman because of her sex based on the assumptions of the comparative 
employment characteristics of women in general; i.e., the assumption that 
the turnover rate among women is higher than among men. 

There is also a prohibition against the refusal to hire an individual 
based on stereotyped characteristics of the sexes; for example, that men 
are less capable of assembling intricate equipment, that women are less 
capable of aggressive salespersonship. The principle of non-discrimina- 
tion requires that individuals be considered on the basis of individual 
capacities and not on the basis of any characteristics generally attribu- 
ted to the group. 

It is also illegal to refuse to hire an individual because of the 
preferences of co-workers, the employer, clients, or customers. When 
state laws are more restrictive on the basis of the sex, i.e., lifing 
weights, hours of work, time before and after childbirths in which employ- 
ment is allowed, such laws areGsOperceeded by this law. 
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VIII GENERAL QUESTIONS 

Before providing a detailed checklist to d^rmine the extent to 
which a program satisfies the criteria detailed above, an institutional 
examination of some general questions would be in or^er. Check (vO the 
answer which is true in your situation. It no one is assigned this task, 
the cooperative program may not be providing students, employers, or 
faculty with enough help. 

1. Who directs and manages the' cooperative education program? 
Counselor 

Financial aid officer 
Instructional faculty 
Placement oipficer 
Student ^employment officer 
Special cooperative officer 
Other 

2. Who generates cooperative jobs? 
Counselor 

Financial aid officer 
Instructional faculty / 
Placement officer 
Student employment officer 
Job fai r ^ 
Students interested in the cooperative program 
Former cooperative students , 
Student employees 
Other 



3. Who evaluates cooperative )ohk in terms of skills a nd Job-re- 
latedness? \ 

Counselor 

Financial aid officer 
Instructional faculty 
Placement officer 
Student employment officer 
Other 

4. How does student learn about the ^cooperative program? 
Counselor 

Financial aid officer 
Instructional faculty 
Placement officer 
Student employment officer 
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Student newspaper 
Other students 
Employer 

Part of campus orientation 

Skills center 

Other 

How does student match his or her vocational interest with 
cooperative jobs? 

Chance 

Interest inventoty 
Computer program 
Talk with employer 
Talk with peers 
Talk with counselor 
Talk with instructor 
Other 

If student was not able to secure a cooperative job, to whom 
can she or he turn for help? 

Counselor 

Financial aid officer 
Instructional faculty 
Placement officer 
Student employment officer 
Other 

To whom can student turn for help during the program? 
Counselor 

Financial aid officer 

Instructional f8culty 

Placement officer 

Student employment officer 

Peers 

Other 

To whom can the student turn for help in evaluation of experience? 
Counselor 

Financial aid officer 
Instructional faculty 
Placement officer 
Peers 
Other 

Where can the student get additional up-to-date information on 
other aspects of this career or related careers? 

School library 
Counsel ing center 
Computer terminal 
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Publ Ic 1 Ibrary 

Occupational information service 

State employment office 

Other 

10, Where can student get information on current Job prospects in 
this career? 

Counseling center 
Instructional f acul ty 
State employment office 
Financial aid office 
Placement office 
Student employment office 
Other 

11 . Who keeps records on success of the cooperative program? 

Cooperative program director 
Counseling staff 
Financial aid staff 
Academic faculty 

Placement staff / 

Student employment staff 

Other 



IX COUNSELING CHECKLIST 

The detailed checklist follows. For instructions on how to use th 
checklist^ see Appendix I. 
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COUNSELING CHECKLIST: RECORDS 



RATING 



PRIORITY 



SCORE 



Statement 



The recoVds of the counseling function: 

!• are^kept by sex, race, ethnicity, handicap, and 
age group. 



2* reveal tfi^e,, percentage of placement with counseling 



and without^cotmsei ing. 



3 



reveal the percentage of attrition from the pro- 
gram with counseling and without counseling. 



\ reveal the satisfaction of students with the job 
\mat,ch witl\ counseling and without counsel ing. 



5* Reveal the ^satisfaction of the employer with the 

. students who had counseling and with those who 
\ did not,'. 



\re\feal the number of employers who continue to 
paryt icijpate in the program. 



Aeve^al the help and publicity ^iven by community 
gfoup^s, emp loyer groups, and advisory groups. 



Y 


1 


N 


? . 




9 


h 


1 


1 




k 




if 
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6 


if 


;) 


1 


k 
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COUNSELING CHECKLIST: STUDENTS 



Statement 



Counselor-'Stuclent Relationship 
The counselor: 

!• has an attitude of respect for the student. 



RATING 



PRIORITY 



SCORE 



provides ways to help students explore their self- 
identity and personal goals; i.e., helps students 
answer questions: Who am I? What are my strengths 
and weaknesses? What do I want out of life? 



3. provides ways to help students learn about career 
clusters. 



explains ladder-approach to careers and the ten- 
dency toward frequency of job change, and job 
mobility. 



5. provides same information on all career potent It 
alitites to both men and women and people of 
minority groups. 



6. has current information on various careers and has 
evaluated it for sex- and race-fairness. 



helps students evaluate cooperative jobs offered 
in terms of skills required. 



8. works with students in developing job attack skill 



9. helps students plan course(s) to take that would 
help fulfill cooperative requirements and their 
vocational objectives. 

10. acts as ombudsperson for the student with the 
administration, the faculty, and the employer. 



11. helps students with problems as they arise during 
the job. 



12. helps students evaluate the'i r experience after 
cooperative program has ended, in terms of perma- 
nent employment optioas and preferences. 



Y S N ? 



3 k \ \ 

k k k k 

,1 k k k 
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»Rat ing 



Priori ty 



Notes : 
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COUNSEL I N6 CHECKL I ST : ADM I N I STRAT I ON 



Statement 



Counselor-Administrat ion Relationship 
The counselor: 

!• helps establish a master schedule for courses and 
registration that will permit equal treatment of 
cooperative students and revise requirements for 
graduation that may be biased against cooperative 
students . 



RATING 



PRIORITY 



5 
c 



SCORE 



2. helps coordinate the schedule of classes with jobs 
during the semester to ensure that course sequences 
are open to "students in the cooperative program. 

3* works with administration in carrying out affirma- 
tive action programs in counseling staff. 



4. helps administration complete compl iance' reviews o{ 
Title IX, vocational education, handicapped educa- 
tion, and other racial and ethnic issues as they 
impact on the cooperative education program. 



has sufficient staff with the right make-up to do a 
proper job. 



trains paraprofessional s and/or peer counselors to 
handle routine or preliminary work. 



1. uses group counseling when appropriate. 



8. coordinates efforts with other counseling programs 
such as VA, rehabilitation, psychiatric, etc. 
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COUNSELING CHECKLIST: FACULTY 



RATING 



PRIORITY 



SCORE 



Statement 



Counse 1 or -Facul ty\ Re 1 a 1 1 onsh I p 



The counselor: 



1 



!• meets with faculty to discuss specific ways courses 
could be modified to meet the needs of cooperative 
students . \ 



2. works with facul^ty to help them understand the need 
for discussing career opportunities in their fields 



3* meets With faculty to train them in sublet ies of 
sexual and racia]\bias that may hold back students 
progress in their \courses. 



k. matches students' feedback with performance objec- 
tives of instructors. 
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COUNSELING CHECKLIST: EMPLOYER 



RATING 



PRIORITY 



Statement 



ih 

o 

o E 



I 



SCORE 



Counselor-Employer Relationship 
The counselor: 

!• meets with employers to understand their needs and 
the needs required on specific jobs. 



2»;^ helps match student? with the employers* needs. 



3. \talks with employers to help them understand the 
wbrkings of the cooperative program. 



Ji^he]ps employers understand ways to evaluate stu- 
dents and coordinate the employer-student evalua- 
tion. 



helps employers understand the laws on equal op- 
portunity and ways to break stereotypes about mi 
nor i tie's and women so they can be promoted and 
trained 'for greater responsibility. 



6. sets up meetings between employers and faculty so 
they can establish a dialogue leading to increased 
recognition of one another *s problems. 



encourages employers to come to campus for job 
fairs and for informal meetings with students in- 
terested in careers in their field. 



8. encourages employers to enter or re-enlist in the 
cooperative program. 
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COUNSELING CHECKLIST: BACKGROUND 



Statement 



RATING 



PRIORITY 



SCORE 



The counselor: 

!• holds a degree in vocational counseling or at least 
one course in vocational counseling. 



Is a member of professional counseling organization 
concerned with vocational guidance. 



3. has paid work experience other than in education. 



is familiar with at least four tests of career 
guidance inventories and understands differences 
in their make-up angi purpose. 



has experience or training in counseling women for 
non-tradi t ional jobs . 



is 'familiar with Title IX of the Education Amend- 
ments of 1972 as it relates to counseling and other 
aspects of the cooperative program. 



is familiar with Title VII of the Civil Rights Act 
as It relates to counseling and jobs. 



IS familiar with the Vocational Education Act of 
1976 (P.L.9i*-i*82). 
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CHAPTER V 
PROGRAM ACTIVITIES 

The. completion pf the counseling checklist will give the program ad- 
ministrator an indication of the strengths and weakness in evaluation of 
counseling and will indicate those areas where improvement could most im- 
pact on the success of the program.^ Other perceptions are necessary', 
however, to give the clearest picture of the evaluation of program effec- 
tiveness. This chapter explains the other features of a cooperative edu- 
cation program and provides a checklist for step-by-step examination of 
Individual program activities which were identified during the needs access- 
ment as high priority concerns on the part of cooperative education program 
directors. 

The activities are divided into two major categories: matching the 
academic program to job requirements; and monitoring student academic and 
job-related progress. 

I MATCHING ACADEMIC PROGRAM TO JOB REQUIREMENTS 

Matching the academic program to job requirements raises the ire of 
many liberal arts faculty members, but it is the essence of a successful 
cooperative education program. 

!• The Work Ethic : The National Institute of Education is doing 
research on the way to mesh career preparation with academic courses and 
job requirements. 'Teople are not prepared for employment or continued 
education because educational experiences that foster realistic career 
attitudes, behaviors and expectations, and general and specific occupa- 
tional skills are often not available,*' NIE researchers state. Many 
people, according to 'uhe director of the U.S. Office ot Career Education, 
Dr. Kenneth Hoyt, do not develop work values: coming to work on time, 
doing one's best, finishing tasks that are begun, and cooperating with 
onci^s f el low workers. 

Thus, it is important that the course instructors help motivate 
tne students to not only learn the skills required on the job but to de- 
velop a sense of the work ethic. 
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2. Credential ing ; Another reason people are not prepared for employ- 
ment Is that their skills, abilities, and competencies are not certified in a 
way useful to continued education or to entry and progression in the ranges 
of occupations for which they may be qualified. 

Willard Wirtz, in a Statement before the Subcommittee on Labor, Health, 
Education, ar.d Welfare of the House Committee on Appropriations, 16 May 197^, 
commented, *'Our credent ia 1 ing processes often constitute barriers to new ca- 
reer paths, promote the lockstep educational pattern from high school to col- 
lege, and prevent many, including women, from re-entering the work force and 
pursuing the careers they desire. Educators and employers must continue to 
seek ways of measuring what a person can do rather than depending on the num- 
ber of years of school achieved and degrees attained.** 

Employers are similarly hanidcapped in sel ect ion because hours of 
course work completed or possession of a degree-based or hour equivalency 
may be poorly related to competencies needed on the job. 

Legal restrictions on job entrance tests make it necessary that the 
training a ;?erson receives will equip that person for a job. It would help 
a student find employment if the employer understood the elements that con- 
stituted the employment training of the individual In the cooperative pro- 
gram. 

3. Academic ski lis : Basic skills are required. Many students who 
have only vocational skills lose out because employers require that a per- 
son be able to read directions and follow them, write up reports on equip- 
ment malfunctions, or make reports on new techniques, workflow, or various 
other elements. It is important that a student who is deficient in basic 
skills receive help to meet basic minimum standards. This is also true of 
language deficiencies. In certain areas, a course in public speaking would 
help the student work more efficiently on the job. Such training might make 
the difference between a dead-end job and one with more advanced potential. 

4. Independence : Students should have experience in structuring 
their own work. Projects may be required in the course of their work to 
help them gain practice In this area. Students should have knowledge about 
the cost-effectiveness of various procedures and methods. Knowledge of 



the theory of hydraulic transmission may bear little relation to the ability 
to ffx a car. In graphic design, for example, It is important that students 
be able to plan a project, and then execute it from start to finish. The 
.successful execution-of a project may be a more effective measure of job 
compcteace than a simple paper and pencil list. 

5* Speed and accuracy ; The speed and neatness with which students 
work comes with practice. However, students may be too hurried to do a 
competent job. In some cases, this may result in gross inaccuracies, or 
important details may be omitted. It is important for students to gain 
enough skills so they can perform with ease on the job. Employers should 
understand the speed at which new employees can perform their tasks so as 
not to put unreasonable demands or pressure on cooperative students. 

6. Safety ; Safety requirements are an important aspect of each 
job. Students should understand what safety problems there are at each 
stage of the job so they C2n take account of them when performing the 
job. For example, the cask of developing film might be detailed along 
with the safety requirements; 

• Load tank in dark room 

• Prepare chemicals for developing and fixing 

• Check temperature of solution and adjust development time 

• Add developer and agitlate chemicals durin>:^ development 

Avoid: 

• Prolonged contact with chemicals 

• Wear rubber gloves and apron 

• Use all solutions at correct temperatures or adjust processing 
time 

• Wear goggles 

• Replace developer with stop bath 

• Place f i Im in drier 

• Wash and dry f i Im. 

7. Legal i ty : Many jobs have laws regarding them. For example, 
in photography, do students understand laWs on model releases, obscene 
photography, counterfeiting, seditious literature, etc.? 

c 

8. Career structure ; Students should understand the entire 
structure of their area of interest. They should be able to identify 
various careers in the field. For example, in printing, the careers might 
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Include linotype operator, intertype or monotype press operator, offset 
press operator, proof press worker, screen maker, silkscreen cutter, silk- 
screen machine operator, and various tindery jobs and stitching jobs. The 
student should have a basic understanding of what each of these people do. 
A student who intends to work in printing should also know what the graphic 
designer and editor do so the student can understand what occurs when chan- 
ges are made or proposals bid on. 

9. Performance-based objectives ; Instructors understanding how to 
write performance-based objectives is a key to the -process of matching the 
course with the job. Performance-based objectives are derived from a 
detailed description of the series of tasks involved in performing a skill 
needed on a job. 

If this process is completed in sufficient detail, an employer could 
examine it and see if there are any serious ommissions or confusions that 
might thwart an individual's performance on the job or even getting a 
Job. 

The director of the cooperative program may provide in-service train- 
ing or a special seminar on writing performance-based objectives. \ 

10. Individual ization : For students to be successful in a course, 
the instructor must take individualization intoaccount. This demands, 
even in a rigidly programmed instruction, customizing it to suit the 
needs of the student. The instructor can serve as a model for the stu- 
dents, as well as a source of encouragement to students who might other- 
wise drop out of school. 

11. Up-to-date instruction : Organization is a major aspect of a suc- 
cessful program. Learning experiences must be structured so that students 
may duplicate them on the Job. In certain newly emerging areas, it may be 
helpful and provide extra interest to have people from industry discuss 
their background and companies. However, it is advisable to discuss the 
content of the presentation with the presenter beforehand. 

Seeing is believing. While a professional teacher or instructor is 
expected to keep up-to-date by reading the llteratJre in her or his field, 
it is necessary to visit work sites where students will receive training. 
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It Is also helpful if the instructor can visit other sites to be able to 
make comparisons and present this information to students. 

It may be possible for the department heads of several cooperative 
education feeder programs to get together to discuss areas of mutual 
concern* In the graphics areas, this might include discussion among the 
photography, communications, graphics, and drafting deaprtments. It may 
be possible for them to sponsor a guest speaker or bring in a special 
display for all students. 

12, Retraining ; It is important that instruction be provided on 
tools and machines similar to those found in industry. If a student 
learns to gain proficiency on certain equipment that is no longer used 
In the field, then he or she is not employable without extensive re- 
training which employerf may be reluctant to provide. 

An institution may hope to cut-corners by using older equipment, 
but this is a mistake. The cooperating employers may be willing to do- 
nate the type of equipment or have students trained in this aspect of 
the course at their plant. 

It may also be possible to get the repair-person of the equipment 
used to come to the class to give a demonstration or answer questions 
about the equipment. When selecting machines or equipment for a course, 
it may be worthwhile for a series of salespeople to make their presenta- 
tions to the class and have the class quiz them on problems they may 
encounter. 

It IS also essential to assess the effectiveness of procedures 
used to monitor che progress of the student through the academic and 
job-related phases of the program; a discussion of this .irea follows. 
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II MONITORING OF STUDENT ACADEMIC AND JOB-REiATED PROGRESS 

P.L. 94-482 requires that programs of vocational education be evalua- 
ted by each state using statistically valid sampling techniques. The law 
further stipulates that programs be evaluated fn terms of the ''extent to 
which completers and leavers (i) find employment in occupations related 
to their training; and (ii) are considered by their employers to be well- 
trained and prepared for '^niployment 

The cooperative education administrator striving to meet the man- 
dates of P.L. 94-482 will sometimes find information readily available. 
At other times, the necessary records, procedures, instruments will 
be much harder to locate, or entirely new systems will have to be de- 
veloped . 

Job-Related Progress 

To assess the procedures used to evaluate job-related progress, the 
program administrator should look for the following information: 

1. The extent to which the program, for completers and leavers, 
(both broken down by age, race, ethnicity, sex, and hanidcap), 
matches 

• the aptitudes of the student with the aptitude required by 
the job 

• the interests of the student with the interests required 
by the job 

• the skills level --academic, technical, and human relations— 
of the student with the entry level skills required by the 
job; and 

2. The extent to which completers and leavers 

• stay on the job 

• get promotions and/or salary increases 

• require retraining, and 

• satisfy the employer. 

Acade-iic Progress 

''o assess the procedures used to evaluate academic progress, the 
prograiT administrator should examine the reliability and validity of the 
instruments used, the content they cover, their cost and ease of use, and 
the collection, analysis, and reporting of data. 

1. Reliability refers to the consistency with which an instrument 
produces information. Types of reliabi 1 ity of interest to cooperative 
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educators are: 

• Equivalence or internal consistency, which measures the homo- 
geneity of test items, the vdegrce to which they measure the 
same skills or underlying chc eristics; 

• Test-retest reliability, which measures the extent to which 
scores remain stable or consistent from one test period to 
the next in the absence of any changes in the examinees; c,nd 

• Alternate-form reliability, which measures how similar exami- 
nees can be expected to perform on different forms of the same 
test. 

2. Validity refers to the accuracy of information provided by an 
Instrument. Types of validity of interest to cooperative educators are: 

• Content validity, which measures the degree to which a test 
covers the objectives and skills it claims to cover; 

• Concurrent validity, which measures the extent to which the 
student's performance on a test is similar to his or her 
performance on some already-validated test that measures 
the same objectives and skills; 

• Predictive validity, which measures the extent to which scores 
on a test are predictive of performance on future tasks; 

• Construct validity, which measures the degree to which scores 
^'^on a test permit inferences about underlying traits or psycho- 
logical traits; 

• Sensitivity to program effects, which measures a test's abili- 
ty to discriminate those who have from those who have not be- 
nefited from a program; and 

• Culture and sex bias, which measures the degree to which a mea- 
sure produces a fair estimate of a particular group's perfor- 
mance. 

Another aspect of validity of interest to cooperative educators is 
norm-referenced and criterion-referenced score interpretations. In an cri- 
terion-referenced test, the score :s given as a percent: the number of 
items a sf^dent answered correctly, divided by the total number of items. 
In a norm-referenced test, the score is given in percentiles, showing how 
well the student did in relation to other students (the "bell curve" method 
of scoring). The groups to which an instrument is normed can be divideH by 
region or state, or by age, race, sex, ethnicity, or handicap. 

.3. In terms of coverage, the instrument should provide the information 
that you need about the student, not irrelevant material. 

4. Failure to account for ease of use and cost may produce unneces- 
sary delays in the start of the evaluation and also could result in a loss 
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of the explanatory powers of its Information, 

5, tn addition to the cost o^ data collection, a number of addi- 
tional factors should be taken into consideration. Data collection is 
a very sensitive issue from the perspective of the faculty and indivi- 
duals administering the program, and from the perspective of individual 
students, due to recent legislation regarding confidentiality and pri- 
vacy rights of those participating in research and evaluation projects. 
In addition, care must be taken to insure that the data collection phase 
does not create undue administrative burdens on those operating the pro- 
gram. To the extent that such individuals are willing to accept certain 
data collection responsibilities, it should be emphasized that the vested ^ 
Interests and subjectivity of data collection participants do not conta- 
minate the data. And finally, once the information is collected, it 

must be managed in such a way' that it is secure so far as privacy is con- 
cerned, yet IS still easily retrievable for analysis and is also flexibly 
categorized such that additional data may be added. 

6. The type of analysis and treatment of the data is directly rela- 
ted to the objectives of the evaluation and/or the requirements imposed 

by funding and/or other agents. It is not the^ purpose of this checklist 
to provide guidance into the types of analyses which could be conducted 
since they are myriad arid varied. There do exist a number of books and 
guides which do provide this kind of assistance, including those prepared 
by Dr. Sarah Steele, Center for Study of Evaluation, UCLA; SPEMS (developed 
by Education TURNKEY Systems); the recent model developed by Fink and Kose- 
coff; and other materials mentioned in Chapter !]• 

Regardless of the types of analysis, there do exist two distinct re- 
porting areas: interim reporting on student progress; and reporting on the 
success of the overall program at its completion. The frequency and nature 
of interim reporting may be a function of the instrument used. For example. 
If the criterion-referenced testing program is being used, then interim re- 
ports could be an accumulation and aggregation of progress indicated on 
the individual students* files. On the other hand, if standardized tests 
are dsedj then alternative forms of ter^ts could be administered periodically 
to a sample of students involved in the program prior to completion. In 
other instances, interim reporting could be based upon criterion tests 
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while final reporting is based upon standardized test scores. Other areas 
Xa which Interim and final student progress can be monitored have been dis- 
^ssed previously (e.g., percentage of placements, salary differentials, 
satisfaction of employers, etc.). 

In conclusion, there are many issues involved in determining the 
merit of procedures for evaluating the match between the academic pro- 
gram and the requirements of the job: the work ethic; credent ia 1 ing ; 
academic skills; independence; speed and accuracy; safety; legality; 
career structure; performance-based objectives; ind ivual izat ion; up-to- 
date instruction; and retraining. 

Instruments used to monitor student academic and an job-related pro- 
gress should provide a profile of the student and of the job, should fol- 
low the on-the-job progress of completers and leavers, should involve re- 
liable and valid measures of academic achievement that are normed to the 
group appropriate for the program, and should use suitable methods of data 
collection, analysis, and reporting. An itemized checklist follows, for 
use as a guide in assessing the degree to which the evaluation effort in a 
particular cooperative education program meets these criteria* 

Ml PROGRAM ACTIVITIES CHECKLIST 

General Instructions : Follow the instructions given for the Coun- 
seling Checklist in the appendix, as the format is the ^same. 

^ Specific Instructions : Make several copies of each of the following 
checklists. Give all pages to the staff of the cooperative education pro- 
gram and ask them to answer all the questions that they can., Then, give 
copies of the Student page to several former cooperative education students 
Let employers answer the Employer page, and instructors the Instructor page 

Compare the answers given by the program staff with the answer pro- 
vided by the students, employers, and instructors. 



PROGRAM ACTIVITIES CHECKLIST: STUDENT 



Statement 



Students who have graduated from the cooperative educa- 
tion program: 

1. know how to use the tools of their trade. 



2, 

T 



know how to operate machines in their field* 



know how to repai r machines in their field. 



RATING 



PRIORITY 



SCORE 



c 
o 



k. read and understand directions on the job. 



2; 
6. 



understand the vocabulary used. 



read and understand technical specifications or 
drawings. 



7. can perform technical calculations in their area, 
to handle problems on the job. 



8. can write descriptions or reports a^ required, 



can perform tasks rapidly enough to keep up on the 
job^ 



10. have enough liberal arts or business background to 
equip them to move into management. 



11. can structure or organize their own wd^rk. 



12. understand the cost-effectiveness of various opera- 
tions. 



13- 

IT. 



can perform with neatness, and/or approp^riate atten- 
tion to ^detai 1 . 



understand safety requirements of the job\. 



15. understand the laws affecting their careei\s, 



16. understand the entire occupational structure in their 
area of interest. \ 



\ 



\ 



Y S N ? 



Rating 




Notes ; 



TALLY TOTALS 
"X WEIGHING FACTOR 
COLUMN SUBTOTAL 



Si gned; 



•73- 



PAGE SUBTOTAL SCORE 



PROGRAM ACTIVITIES CHECKLIST: INSTRUCTORS 


<^RAT!NG 


PRIORITY 1 


1 SCORE 


Statement 
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Instructors In the cooperative education program: 




























1. 


understand how to write performace-based object- 
1 1 ves • 




























2. 


describe courses in cooperative program in terms 
of skill objectives. 


















> 










3. 


state the conditions under which, or tools with 
which, students perform their skills. 




























A. 


state the acceptable level of performance in 
thei r objectives. 




























5. 


break skills down into specific tasks involved 
in doing that job. 




























6. 


compile task lists by talking with individuals 
now doing the work, watching them on the job, and 
talking with their supervisors. 




























7. 


list tasks along with the frequency of their per- 
formance, their inportance, and the learning dif- 
ficulty involved. 




























8. 


arrange the task list in order of sequence that 
must be fol lowed. 




























9. 


list 
task. 


tools or materials needed to perform each 




























10. 


list 
each 


safety considerations or hazards involved in 
step. 




























11. 


provide practice in tasks under conditions similar 
to those found on jobs. 




























12. 


allocate sufficient time to practice skills so 
they can determine when task is done wrong. 




























13. 


give 
lists 


students a cooy of the objectives and task 




























1*». 


arrange course(s) so as to take student ability 
into account. 




























15. 


help students in areas of weakness or deficiency. 




























lb. 


arrange learning experiences in orderly sequence. 
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PROGRAM ACTIVITIES CHECKLIST: INSTRUCTORS (cont'd.) 



RATING 



PRIORITY 



SCOR 



Statement 



^1 



cl 



17* bring in experts from the field to demonstrate 
techniques or discuss problem areas. 



18. regularly visit the industry in their professional 
area. 



19. plan field trips to industry or business sites 
where students can see actual operation. 



20. have paid experience in the area they are 
teaching. 



21. have formal training in the area they are 
teaching. 



22. keep in touch with faculty members to let them 
know what they are doing to help them alert stu- 
dents to overlapping areas of Interest that their 
students may wish to sit in on. 



23* read professional journals and attend professional 
meetings to keep up-to-date in their area. 
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^PROGRAM ACTIVITIES CHECKLIST: EMPLOYERS 



RATING 



PRIORITY 



SCORE 



Statement 



Employers of cooperative education students: 
U participate in program planning sessions. 



2* identify skills needed for success on the 
job in-specific skill terms and in human 
relations terms. 



3. visit campus to meet wjth academic instructors, 
^director of the cooperative program, anH ad- 
ministration. 



ptjovide samples of materials used or problems 
found in their jobs. 



5. provide work experience for faculty members 
. during summer. 



6. structure cooperative jobs o they lead to 
upward mobi 1 i ty . 



7. seriously consider the terms of the coopera- 
tive contract. 



8. use students' in a^eas of students' interest. 



.3^ are aware of students' differing learning 
styles. 



10. agree to specific ways to evaluate students 
-and communicate methods to be used on students^ 
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PROGRAM ACT I Vi TIES CHECKLIST; DIRECTOR 



RATING 



Stateme.it 



The cooperative education director: 

1. requires a list of performance objectives from 



instructors . 



2. shares these objectives with employer. 

3. gets feedback from employer. 



gets performance objectives from employer. 



5. 'shares these objectives with faculty and dis- 
. cusses areas o? nonconjunction to find ways 

to overcome problems. 

6. checks with employers at end of semester to 
see if the course work has fulfilled the 
employee;s needs and to discover how it might 
be furt er modified or changed. 



7. discusses affirmative action programs with 
employers. 



or t'-e counselor monitors industry to see that 
they are doing what they said they would. 



9. ensures that employers provide more than me- 
nial tasks for students. 
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PROGRAM ACTIVITIES CHECKLIST; ISSUES 



Statement 



The instruments/recfords/procedures used in the 
cooperative education program provide information 
about the 

!• assignment of students to cooperative education 
programs « 



RATING 



2, program leavers as well as completers. 



3. student's age, race, sex, ethnicit/, and han- 
dicap. 



stud€tnt*s previous academic training. 



5. student's job training courses. 



6. student's job experience while in school 



7. student's aptitude. 



8. aptitude required by the job 



comparison of the aptitude of ihe student with 
the aptitude required by the job. 



10. student's interests, 



11. interest? required by the JcD. 



12. comparison of the interests of the student and 
the interests required by the job. 



13. academic level of the student. 



1^. entry level academic skills required by the 
job. 



15. comparison of the academic level of the student 
and the entry level academic skills required 
by the job. 



16. academic courses offered in the program. 



17. technjcal skill of the student, 



18. entry level technical skill required by the 
job. 



19. comparison of the technical skill of the stu- 
dent and the entry levil technical skill re- 
qui red by the job. 



20. job- real ted courses offeree in the program. 




Notes: 



PRIORITY 



SCORE 



TALLY TOTALS 
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COLUMN SUBTOTAL 
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PROGRAM ACTIV|t;eS CHECKLIST: ISSUES (Cont'd.) 



Statement 



2K human relation^, skill of the student. 



22* entry level human relations skill required by 
the job. 



23. comparison of the human relations skill of the 
student and the entry level human relations . 
skill required by the job. 



RATING 



PRIORITY 



SCORE 



cj 
-^1 



2^. duration of completers on the job compared to 
the duration of leavers on the job. 



25. duration of students compared to non-students. 



26, 



salary of completers after 6 months compared 
to the salary of leavers after 6 months. 



salary of students compared to non-studenis 
after 6 months. 



28. salary of completers after 12 months compared 
to the salary of leavers after 12 months. 



29. 

lo^ 



salary of students com^.;.^d to non-students 
after 1 2 months . 



salary of completers after I8 months compared 
to the salary of leavers after l8 months. 



31 



salary of stuHents compared to non-students 
after 1 8 montns . 



32* on-the-job retraining needed by completers 
compared to on-the-job retraining needed by 
leavers . 



33, retraining needed by students compared to non' 
students. 



3^. employer satisfaction with completers compared 
to leavers after 6 months. 



35. employer satisfaction with students compaiad 
to non-students after 6 months. 



36. employer satisfaction with comp>eters compared 
to leavers after 12 months. 



37, employer satisfaction with students compared 
to non-students after 12 jnonths. 
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PROGRAM ACTIVITIES CHECKLIST: ISSUES (Cont"d.) 



RATING 



PRIORITY 



SCORE 



Statement 



3 
O 
c 



c 
o 



38* employer satifaction with completers compared 
to leavers after 18 months. 



39» employer satisfaction with students compared 
to non-students after 18 months. 



^0. completers who hold jobs in their area of in- 
terest after college compared to leavers. 



students viho hole' jobs in their area of in- 
terest after college compared to non-students, 
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PROGRAM ACTIVITIES CHECKL 1ST: IN 5TRUMENTS 



Statement 



The testing instrument used in the cooperative 
education program; 

K clearly identifies the subject areas measured 



measures the subject areas of most conern to 
the program. 



RATING 



PRIORITY 



SCORE 
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c 
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3. costs less than $1.00 per student to buy, 



k. costs less than $1.00 per student to administer 
and score. 



f>. records information for each student, 



6. records information for each clasrroom. 

7. records information for each program. 



8. gives complete directions for its use. 



9. requires no special equipment, 



10. can be completed by a student in less than 
30 minutes. 



11. C3n be completed by a student in less than 
60 minutes. 



i2. takes more than an hour to complete. 



13. 



is to be completed by school staff. 



is to be completed by spec i 1 1 i sts . 



(S. uses an objective scoring system. 



16. uses a score interpretation formula that is 
easy to unde stand . 



17. reports scores by individual subject areas 



18. reports a single undivided score for each 
student . 



19. contains provisions for norm-referenced score 
interpretat ions . 



20. 



provides norms for different ethnic groups. 



21. provides norms for males and fenidles. 



22. provides nationally and regionally represen- 
tative nornr.r. 



Y 


s 


H 


? . 




S 


k 


1 


1 




k 


k 


k 


4 


'\ 


1 


k 


6 


k 


Nil 


1 


k 


ii 




V 



'Ration 



Priori ty 



Notes : 



TALLY TOTALS 
X WEIGHING FACTOR 
= COLUMN SUBTOTAl 



PAGE SUBTOTAL SCORE 



C T MM • 



PROGRAM ACTIVITIES CHECKLIST; INSTRUMENTS (Cont^d) 



Statement 



23* provides geographically restricted norms 
(e.g.> a single district). 



24* provides for criterion-referenced score inter- 
pretations. 



25. sets criteria on the basis of empirical data. 



26, provides useful inforniatlon about its own 
val idl ty. 



27. 



28. 

"29^ 



provides scores that can only be interpreted 

by a special ist , 

provides scores That can be interpreted by 
school personnel 



RATING 



PRIORITY 



SCORE 



2 
o 
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provides scores that can be interpreted by 
the student. 



30* provides a complicated score interpretation 
gu ide. 



31. provides uncomplicated score interpretation 
gui de. 



32, 



33- 

1?: 



provides limited guidelines for decision- 

making based on the student's scores. 

provides extensive guidelines for decision- 
making based on the student's scores. 



provides a single test score. 



35. 

1^ 



provides a few subscores. 



provides many subscores. 



37* provides a profile with no scores. 
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PROGRAM ACTtVITIFS CHgCKLtST: DATA milFrTinN 



RATING 



PRIORITY 



SCORE 



Statement 
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The data collection techniques used in the cooper- 
ative education program: 

!• minimize disruption within the program during 
the data collection phase. 



ensure the protection of individual privacy 
rights and confident ial i ty. 



minimize the possibility of subjectivity on 
the part of those collecting the data. 



allow for easy reduction in accordance with the 
data management plan. 



ensure that data can be easily retrieved for 
analysis purposes in a pre-determined format. 



provide flexibility for updating and/or modi 
fication of the data base. 
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PROGRAM ACTIVITIES CHECKLIST: DATA ANALYSIS 



RATING 



PRIORITY 



SCORE 



Statement 



The procedures for data analysis and reporting in 
the cooperative education program: 

are consistent with the overall evaluation ob- 



jecttves and external requirements. 



2. provide for interim progress reporting on aca- 
Jemic achievement. 



3. provide for interim progress reporting on on- 
the-job training progress. 



h. provide opportunities for feedback to instruc- 
tors and the cooperative education director. 



5. provide information in a useful format, in a 
timely and efficient manner. 



6. provide for the use of statistical applica- 
tions to treat the data where appropriate. 
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CHAPTER VI 
RELATING PERFORMANCE AND COST 



Most education legislation over the last five years has explicitly 
or implicitly required local and stal.: education agencies to demonstrate 
the relationship between the cost of programs and their success. This Is 
particularly true in the new Education Amendemnts of I976 affecting coop- 
erative education. 



I GLOBAL ISSUES 

Virtually all citizens in this country^ conciously or unconciously, 
relate cost "o performance or benefits daily in decisions they make. In- 
deed, the backbone of a '^free" enterprise market mechanism is the constant 
weighing of these two factors. However, while these criteria are upperr 
most in the minds of decision-fnnkers in the private sector, consideration 
of these factors in education and ocher government-operated programs is 
often a lower, sometimes clouded priority for a number of reasons. 

First, unlike measuring profits in a private sector, no such "bottom 
line" exists in governmental operations and in education. The notion of 
measuring the productivity of government continues to conjure up political 
debates and consume considerable amounts of the resources allocated for 
education research and developnent. 

Second, unlike many consumer goods purchased over the counter, educa- 
tion is also perceived to be an investment in "human" capital. However, 
given the increasing technological changes affecting the social and econo- 
mic milieu in which people increasingly gain horizontal and vertical mobi- 
lity, it is indeed difficult to determine what portion of one's education 
can be considered an investment and what portion a consumable item. Per- 
haps over time, with the emergence of concepts such as career education 
and leisure education, the distinction will become more clear. 

Third, determining the cost and benefits of education in our society 
is also a debatable topic, and without a clear determination of either, a 
proper comparison cannot be made. While the value of education to an in- 
dividual may be determined, the value to a society having well-educated 



citizens is less discernable except in a general sense. On the other hand, 
the high cost of education can be identified. The additional cost to so- 
ciety of not having well-educated citizens is less easily determined. How- 
ever, at the national level, these types of cost-benefit relationships have 
to be taken into account for policy-making purposes which attempt to dis- 
tribute and redistribute funds for educational purposes. 

il TECHNICAU ISSUES 

As one focuses upon the relationship of cost to performance in educa- 
tional institutions, a number of technical issues immediately arise in ad- 
dition to those more global issues described above. First, of primary con- 
cern Is the criteria for measuring the success of programs. These crite- 
ria generally fall into two categories: (l) those measuring various types 
of outputs or outcomesi and (2) those measuring the processes of delivery 
of services provided. 

Criteria for Keesuring Performance of Programs 

1. Outcome critr-ja: Outcome criteria reflect or should reflect 
perceived omissions, goals and objectives of programs. For example, if 
one considers the primary function of public schools to be that of a cus- 
todian for a period of a child's life as required by compulsory attendance 
laws, then the criteria for success could include: (l) maintaining the 
scheduled operations of schools; (2) maintaining the health and physical 
well-being of students; and (3) ensuring the maximum number of school age 
children actually attend schools. If on the ether hand, one perceives the 
major objective of school operations to impart skills in areas such as 
reading, then the primary criteria of success could lange from norm-refer- 
enced, standardized test scores to individually-designed performance- 
based measures such as comprehending drivers' manuals, newspapers, etc. 

In addition to the above surrogates of performance, several longer- 
term measures can be used to assess programs. If on one hand, the ob- 
jective of participating in an education program is to learn for the sake 
of learning, then the ultimate goal is whether or not the individual is 
"happier'^ or generally ^'satisfied" after receiving such education. If on 
the ocher hand, the education or training is considered more of an invest- 
ment Item where the individual or the state is expecting some return on 
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that Investmenti then measures in some instances can be more specific, such 
as: (1) probability of employment for completers and leavers of the pro- 
gram; (2) retentipn of the job after a period of time; and (.3) salary dif- 
ferential between \those receiving training and those who did not. Hence, 
in selecting the speci f ic cri ter ia for assessing program outcomes or out- 
puts one should take into account not only short-term criteria but also 
more long-term criteria, both of which have to be questioned in terms of 
their accuracy, reliability and even validity. 

2. Delivery of services: The other set of criteria relates to the 
delivery of service*^ as a measure of success. In education programs, par- 
ticularly those that are operated as special projects outside of regular 
program operations, those individuals managing these projects usually have 
a pr<. determined plan which specifies all of the ta^ks and activities which 
have to be completed before the project itself is completed. The criteria 
for success in this case 55 the degree to which the plan is impiemented 
in terms of several criteria, including time, cost, and l^vel of perfor- 
mance. In most education projects. hc.;ever, the operat5on«^l time is u- 
sually prsdetermined (i.e., schools opercte ISO day; per scl^icol year, and 
programs operate for a semester, etc.), and the amount of funds is usually 
predetermined, especially when provided by an outside agency. Since there 
exist few, if any, incentives to conduct the project without spending the 
total amount budgeted, the major criterion is usually the level of per- 
formance achieved by the services. Hence, the criterion for the manager 
of the project is usually to maximize the performance levels of services 
provided within a fixed time constraint and given a fixed budget. In edu- 
cation programs which are not, however, funded through outside sources nor 
are considered to be special projects but father part of an overall '-egular 
program operation, the constraints of time apd budget are often relaxed, 
in which case the manager is constantly trading off time against cos' 
against performance in meeting the overall objectives of the project (e.g., 
"If I descrease time the cosus will ,drop; but what happens to the level 
of performance?"). Even in programs of this nature, flexibility seldom 
exists. 
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Costs/Resources 

A second set of issues relate to cost. When one addresses the ques- 
tion of cost in education programs there is a natural tendency to equate 
costs with what is reflected in budgets. The preparation of accurate bud- 
gets is usual ly^^ucial at the beginning and end of projects, A budget 
projecting resources to be used is usually required to obtain funds, or 
is<yrequired by auditors the completion of projects to ensure that all 
funds are properly accounted for. Except in the most e>rstreme cases, bud- 
gets — either projected or actual — tell very little about program ef 
ficiency and are virtually useless in allowing one to relate cost to per^ 
formance. Budgets do, however, provide a starting point for identifying 
cost relationships. 

Increasingly, resource consumption models and techniques arc* being 
used to relate cost to performance. Usually in these resource consump- 
tion models, the total amount of resources by type of resource (facility, 
Svaff, etc.) is prorated (as consumed) to functions or specific programs 
(e.g., reading programs, training programs, etc.). For example, a budget 
might indicate the following subtotals for the entire program: instruc- 
tional salaries; administrative salaries; custodial salary; equipment; sup 
olies; and services. In contrast, a resource allocation model indicates 
whar. slice of each of these budget categories applies to each part of the 
program. For Instance, it would indicate wh^st percentage of the instruc- 
tional salaries budget, and of the administrative salaries budget, and of 
the supplies budget, etc., is allocated to the reading program. 

Exhioit IV-1 illustrates a resource consumption cost model (i.e., 
the COST-ED Model) for identifyir costs, relating costs to functions, 
and eventually to performance le^el achieved by those fu..ctfons or pro- 
grams . 

1. Staffing: !n virtually all types of education programs, the 
most sensitive and critical cost factor is staff time, and as a result 
the most critical element in a good resource consumption model is the 
ability to identify time usage patterns by various staff members and the 
proper proration of that time to specific programs and functions. 

Standard and Actual Pricing: A second major issue related to 
to cost, particularly in state-wide and nations! evaluations, is whether 
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to use standard pricing or actual pricing for the types of resources (aga»n, 
mostly staff) which are used. One problem that arises is the wide variance 
In, for example, teacher salary when comparing a metropolitan area to a 
rural area. One can standardize for these variances^ in studies involving 
a national sample, for example, by using the average salary rate for 
teachers with various levels of experience and seniority, rather than using 
actual prices. On the other hand, it might be advisable, in studies of 
a different nature, to use actual prices for the following reasons. First, 
by using the actual prices of a locality rather than standard prices, one 
is taking Into account the fact that efficiency criteria might be used by 
planners at the experience level. For example, in designing a remedial 
readfng program where- available instructors have a large number of years 
experience (hence, high salaries), and where peer-tutors are paid only 
the minimum wage, a program planner could conceivably decide 'that three 
tutors would benefit the program more than the addition of one certified 
experienced instructor, even though the cost of the two alternatives would 
be the same. If one were to use standard pricing in comparing the cost 
and performance of this program to a national standard where the cost of 
aides was 50% higher, the relative cost of this program in comparison to 
others would be less cost-effective than if one used actual pricing. 

Generally speaking, standard pricing is probably more advantageous 
In ^hatlonal or state-wide studies where the sample is supposed to be re- 
presentative. On the other hand, actual pricing is more likely to be 
recommended when the analysis is closer. to the local institutional level 
or where specific programs are compared to each other, even on a national 
basis. 

3. 'Tree*' Resources: Third, another technical issue is the manner in 
which **free** resources are treated. This is particularly true In educa- 
tion programs which have workstudy or community oriented education compo- 
nents, where resources are provided at no cost or are funded through 
other governmental agencies. The treatment of these resources Is directly 
related to the perception of the specific institution doing the evaluation. 
For example, from a national perspective, free resources for the most 
part should be considered part of the total operating costs and should be 
allocated to the particular functions of programs. However, at the 
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local program level, from a management point of view, free resources are 
normally not included in any analysis. However, if one conducts a longi- 
tudinal study where the availability of free resources varies over time or 
a comparative study comparing one program to another over time, then 
such resources may be required for inclusion in the analysis. Again, de- 
pending upon the perception of the operating agency and its philosophy, 
these costs are often excluded for several reasons, including: (l) the 
difficulty of Identifying such cpsts with any degree of accuracy; and 
(2) the assumption th^t the participating agency is benefiting to a 
level equal to the cost it expends in which case such resources are 
assumed away from the analysis. 

Joint Users: And last, a cost consideration sometimes arises when 
certain costs, usually fixed costs, are incurred by joint users (e.g., the 
cost of a computer-based counseling service). The identification and 
determination of joint costs and then their proration is a methodological 
issue which has to take into account several factors. First, if the 
educatmn program Is of a long-term duration, then the major concern from 
the management perspective is the marginal cost or variable cost which 

could change over time. If on the other hand, the program is of a short 

ft 

term duration then it may be necessary to include as part of the overall ' 
operating costs a proration of the| joint cost which Is directly related 
to either support or operating functions within the education program. 
Second, the degree to which the ecjucation program manager has control 
over the resources often determines the method by which one identifies and- 
al locates or prorates joint costsj For example, over the last decade a 
large number of educational regional service centers have been created 
within states which provide computer services to education program managers 
at the LEA level. To the extent that these services are provided through 
contractual obligations, the user costs are relatively easy to identify; 
and a joint cost can be more easily identified. On the other hand, however 
when such resources are made available at low priority times, at varying 
degrees of quality, and through informal arrangement, the opposite is 
true. 
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Relating Performance and Cost 

A third set of technical issues relate to the technique for analy- 
zing and attributing resources consumed to performance and other outcome 
measures. Ideally, one would want to be able to develop an overall model 
based on historical data which would allow one to project the marginal 
increase or decrease in performance by reallocating existing resources 
or adding resources to particular programs or functional areas. For 
instance, such an ideal model would answer the following question: to 
increase placements in computer-related jobs by, 10%, should instructional 
salaries In computer-related courses be increased?; and by what percent?; 
or should counseling salaries be increased, and by what percent?; of shodld 
instructional expenditures be increased?; and by what percent? 

While this may be possible in certain areas such as defense, aerospace, 
etc., where physical laws have been observed (e.g., the law of thermodyna- 



lIcsX,_aod.-can_be-appl-iecl-to capital Hntens-Ive- funct1ons7 th~e state of 
the art in education does not at the present time permit It except at a 
very gross level. At the present time, the best one can do is to identify 
process variables related to organization, management, and Instruction, and 
to some degree corrolate these to various outcome measures. Once this is 
done, then one can identify the amount of resources consumed by these pro- 
cesses or implementation variables. For example. It Is possible to Iden- 
tify factors°corre1ated with redemial reading achievement, such as program 
monitoring, number of In-service training days, etc., and the costs of im- 
plementing each of these processes. However, the specific causal rela- 
tionships remain unclear. 

Relating performance and cos>. may involve new ways of data collection, 
such as a new requirement for admi\ii strati ve staff to fill out time sheets 
listing hours per day spent on vari'ous programs; but in the long run, the 
benefit to the program of being able^ to relate performance to cost will 
outweigh the initial inconvenience. 

Once the program administrator h^as taken these major technical issues 
criteria for measuring the performance of programs; costs/resources; 
and relating performance and cost — into consideration, he or she will 
be able to more effectively manage progi*am resources to produce the de- 
sired outcomes. 
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in GENERAL QUEST IONS 

Before going through the checklist designed to assist in relating 
performance to cost, the program administrator should address a few 
general questions. Place an X by all answers that apply, and fill in 
the blanks where appropriate to the program. 

1. Whet are the specif ic^criterla used to judge the success of your 
cooperative education program? 

Continued employment of completers after months on the job. 

Salary of completer at beginning and months after employment. 

Number of participants completing progran. 
Achievement of participants in academic program. 
- Demonstrated increase in skills and/or levels of performance as 

result of participation. - 

Satis fact i on_of.-emp 1 oye r- a f te t — "^n iTontH s of employment. 
: Othe r 

2. What will be the unit of analysis for reporting on the impact of the 
cooperativeT education program? 

Individual participant 

Portion of participants in specific programs 

Performance of all individuals* in specific programs 

All participants and all cooperative education programs 

Other ^ , 

3. Please describe how the unit of analysis will be aggregated: what type 
of statements will be made? (e.g., "80^ of participants are on the job 
three months after completion of the program and entry into the work 
force.'*): ' 



k. Identify all of the resources used and functions/services provided for 
each of the cooperative education programs to be analyzed. Do so by 
identifying those funds consumed by participants in each of the functional 
areas described below by putting a dollar figure ^in the appropriate box 
(see Exhibit IV-2, following. For Instance determine how many hours the 
Instructional staff spends per year doing counseling; then multiply that 
by the average hourly rate for instructional staff; and put the results 
In the top left box of Exhibit IV-2. 
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RESOURCE CONSUMPTION 



Resource 



Instructional Staff 

Staff Assistant 

AdmrnrtsT ra tTveTtaf f 
Clerical/Secretary 
Counseling Staff 
Job Develop. Staff 
Placement Staff 
Instructional Staff 
Amlnlstrative Facilities 
Data Processing Services 
Reproduction 
Books & Software 
Instructional Equipment 
Funrniture 

Other Supplies & Matls. 
Transportation Services 
Other 



Total Yearly Cost Per 
Parti cipant 

Total Yearly Cost Per 
Component 
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IV CHECKLIST 

The purpose of the following detailed checklist Is to assist the coop- 
erative education program director In assessing existing procedures of re- 
lating performance to cost. Applying this checklist to existing procedures 
combined with the preceding General Questions, should ensure that all fac- 
tors have been considered or. In certain cases, could provide guidel Ines 
for developing techniques for data collection, analysis, and reporting. 
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PERFORMANCE & COST: RESOURCE ALLOCATION 



RATING 



PRIORITY 



SCORE 



Statement 



5J 

cj 



Time usage patterns of staff have been iden- 
tified and prorated according to program 
components, using data of records regarding 
staff salary a^d prices. 

Total yearly cost per participant per program 
component has been derived from the steps in- 
volved In filling in Exhibit VI-2. 

^11 sources of resource information have been 
identified: . 

budget fcr salaries, fringes 

insurance policy for replacement 
orders 

purcTiase orders, vouchers. In- <• 
vo i ces 

schedules for usage patterns 

program planning documents other 
than budgets 



Y 


S 


N 


? . 




■> Rating 


9 


4 


1. 


1 






k 


4 


4 


4 




^ Priority 


1 


4 






Nl' 


1 


1 


4 


4 


4 


7> 
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Notes : 



TALLY TOTALS 
X WEIGHING FACTOR 
» COLUMN SUBTOTAL 



Signed; 
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PAGE SUBTOTAL SCORE 



Date: 



PERFORMANCE S COST: ANALYSIS 



RATING 



PRIORITY 



SCORE 



Statement 



(0 

s: 
O 

CO 



The unit, on which cost, is reported is the 
same as the unit on which performance is 
reported (for instance, if effectiveness 
Is judged by numbers of students completing 
the program, then the unit for reporting 
cost Is average cost per completer). 

The design allows for changes in the unit of 
analysis (from student/level, to class/level, 
to program component/level, to prog ram/ 1 eve 1 ) 
to ensure compatabi 1 ity with the unit on 
which cost is reported. 

The analysis allows correlation of process 
with performance (for Instance; it is pos- 
sible to relate the planning and direction 
of the counseling component to the success 
of counsel ing) • 

The analysis also identifies the resources 
consumed in various processes such as plan- 
ning, training, etc. 




Rating 



Priority 



Signed; 



Notes : 



TALLY TOTALS 
X WEIGHING FACTOR 
« COLUMN SUBTOTAL 
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PAGE SUBTOTAL SCORE 



Date: 
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APPENDIX 
HOW TO USE THE CHECKLISTS 
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APPENDIX 

. \ 

HOW TO USE THE CHECKLISTS 



Now that the broad questions a re' answered, the program administra- 
tor can begin a. detailed evaluatFon using the Counseling Checklist. This 
and other checklists may at first seem time-consuming but will In the long 
run save wasted effort, because they reveal where energies are misdirected: 
important Items that are neglected and unimportant things that are receiving 
too much attention. They also point out the areas of Ignorance about the 
evaluation system that are most crucial to change. 6 

By following the directions below the program administrator can 
learn how to use the checklists with little or no further training. 

STEP K Make several coprjes of the checklist; keep the original 
for further copying as needs arise, 
c .STEP 2 . For-each statement in the checklist, make three, marks In 
the columns at the right hand side of the paper. 
• Under RATING place a check mark (>/) under 

YES If the statement is true 677100% of the time 
SOMEWHAT if the statement is true 3^-66% of the time 
NO if the statement is true 0-33% of the time 
DON'T KNOW if there is no information about the truth 
of the statement 
For example : 



COUNS ELING CHECKLIST; STUDENT 



RATING 



PRIORITY 



S.COR 



Statement 



The Counselo r; 

1. Has an attitude of respect for the student, 



g 



(This answer indicates that, according to available information, in some 
cases the counselor's attitude reflects respect for the student.) 
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Under PRIORITY place a check mark ( ^) under 
IMPORTANT if the statenient is crucial to the success 
of your program 

SOMEWHAT if the statement affects the success of your 
program in some way 

NOT IMPORTANT if the statement has no affect on the 
success of your program j 
DON'T KNOW if the impact of the statement on your program / 

l.s not known / 



For example; 



/ 



/ 



COUNSELING CHECKLIST: STUDENTS, 



h 



Statement 



RATING 



3 
O 
c 



c 

O 



PRIORITY 



SCOR 



•9 



The Counselor ; 

1. Has an attitude of respect for the student • 



(This answer Indixates that, in the opinion of the program administrator. 
It is crucial to the success of the program for the counselor to have 
an attitude of respect toward the students.) 
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COUNSELING CHECKLIST: STUDENTS 



Statement 



Under SCORE, place a check mark ( \/) under 

9 If you already checked YES and IMPORTANT 

6 if you already checked NO and NOT IMPORTANT, or 

1 if you already checked 

• YES and NOT IMPORTANT or 

• NO and IMPORTANT^ 

• DON'T KNOW and IMPORTANT or 

• YES and DON'T KNOW 

!» If you already checked any other combination 
(for convenience these directions are summarized 
In the matrix at the left and on the checklist 
pages ) 
For example ; 



The Counselor ; 

1. Has an attitude of respect for the student. 



RATING 



PRIORITY 



SCORE 
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(The combination already checked in this example Is SOMEWHAT and IMPORTANT, which, by 
the above definitions scores a "A".) 

STEP 3 . At the bottom of the page, place the tally of all check marks in each 
of the SCORE columns in the spaces provided after the words TALLY TOTALS; For example ; 
/ 
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2, 'Ways ^re provided to help students explore their 
self-ldenty and personal goals. 



fl Ways are provided to help students learn about 
career clusters. 



51 Counselor explains ladder approach to careers and the 
tendency toward frequency of job change. 



V 



(Counting the example at the top of the page, there 
Is now one check mark ( \/) iii each column, so the 
TALLY TOTAL in each column is "1".) 



y 



TALLY TOTALS 
X WEIGHING FACTOR 
.» COLUMN SUBTOTAL 



1 



PAGE SUBTOTAL SCORE 



113 

-106- 



STEP Multiply each TALLY TOTAL by the weighting factor below it to 
derive the column sub-totals; then add the column sub-totals together 
to derive the page subtotal score of the program evaluation effort. 
For example; 




Signed; 



Notes; ' 



TALLY TOTALS 
X WEIGHING FACTOR 
.» COLUMN SUBTOTAL 



1 



PAGE SUBTOTAL SCORE 



Date: 



20 



fluontlnuing the same example, there was a tally total of "1" In each colunn. 
"1" times the weighting factor of "1" is "1"; "1" times the weighting factor 
of "A" is "4" and so. forth. ) 

STEP 5 . Copy the SCORE SHEET FOR COOPERATIVE EDUCATION PROGRAM EVALUATION . 
(exhibit 1) and/ save the original for further copying a's needed. 
STEP 6 . Add the page subtotals together to derive the Checklist Score, and 
place the Checklist Score on the SCORE SHEET FOR COOPERATIVE EDUCATION 
•PROGRAM EVALUATION (See Exhibit 1). 

STEP 7 . Add the Checklist Scores together to derive the Program Evaluation 
Score. Use this score as base figure against which to compare scores 
resulting f rom .di f ferent methods of program evaluation, or proposed changes 
In techniques or reports. The higher the score, the better the proposed 
change meets the needs of the programs. 

STEP 8 , Use the checklist pages to guide efforts to improve evaluation. 
Concentrate first on getting rid of the "1's", bec'ause they indicate where 
energy is v-.urrently being misdirected. 
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EXHIBIT 1 
SCORE SHEET 
COOPERATIVE EDUCATION PROGRAM EVALUATION 



I Checklist 1 



\% increase 
lover last mo. 



Checklist 2 



% increase 
over last md. 



Checklist 3 



% increase 
over last mo- 
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